Large model illusion detection method and device, electronic equipment and storage medium

Abstract

The invention discloses a large model illusion detection method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the symbol segmentation and semantic segmentation of the output content of a large model, obtaining a plurality of source sentences and sub-propositions, and determining the sub-propositions corresponding to the source sentences; performing knowledge extraction on the to-be-analyzed file to obtain a reference material; aiming at each source sentence, searching a plurality of reference material elements most similar to the semantics of the source sentence from the reference material; judging whether the searched reference material elements are contained in the source sentence or not; if yes, outputting a detection result of the source sentence; if it is judged that the reference material elements do not exist or are not found out, for each sub-proposition corresponding to the source sentence, multiple reference material elements with the maximum semantic similarity of the sub-propositions are found out; carrying out implication judgment on the sub-propositions by utilizing the reference material elements, and determining detection results of the sub-propositions; and counting the detection results of the source sentences and the sub-propositions, and determining the illusion detection result of the large model.

Description

Technical Field

[0001] The present application relates to the field of model detection technology, and in particular to a large model hallucination detection method and device, electronic equipment, and storage medium. Background Art

[0002] Large language models are usually referred to as large models. They have shown good results in many fields. However, the essence of large models is probabilistic models, so it is inevitable that there will be hallucinations in the output results, that is, the output result sentences are fluent, but they are meaningless, that is, they are meaningless expressions, or they are inconsistent with the reference materials, that is, they contain false information. Therefore, hallucination detection of large models is an important task to detect the quality of large models.

[0003] One way to detect hallucinations in large models is to use search engines or external reference materials to directly compare the output results of the large model to determine whether the output results are inconsistent with the content of the reference materials, thereby determining whether the output results are hallucinations. Another way is to determine whether there is a hallucination based on the probability value of the output results of the large model, or based on the matching similarity of multiple output results of the large model for the same problem. If the probability value of the output result is low, or the similarity of multiple output results is large, it is considered that a hallucination has occurred.

[0004] However, the existing methods are mainly applied to English output results, and lack detection methods applied to Chinese output results. In addition, since English expressions are relatively simple and clear, accurate detection results can be obtained by using the existing methods. However, Chinese expressions have many meanings, and there are multiple expressions of the same meaning. Therefore, directly reusing the existing methods cannot guarantee accurate detection results. Summary of the invention

[0005] Based on the above-mentioned deficiencies in the prior art, the present application provides a large model hallucination detection method and device, electronic device, and storage medium to solve the problem that the prior art lacks a method for accurately detecting hallucinations in Chinese output content of large models.

[0006] In order to achieve the above objectives, this application provides the following technical solutions:

[0007] The first aspect of the present application provides a large model hallucination detection method, comprising:

[0008] Performing symbol segmentation on the output content of the large model to obtain multiple source sentences, and performing semantic segmentation on the output content of the large model to obtain multiple sub-propositions;

[0009] Tracing the source of each of the sub-propositions to obtain the sub-propositions corresponding to each of the source sentences;

[0010] Extract triplet knowledge from the document to be analyzed to obtain reference materials;

[0011] For each of the source sentences, searching for multiple similar reference material elements corresponding to the source sentence from the reference material; wherein the similar reference material corresponding to the source sentence refers to a reference material element whose semantic similarity with the source sentence meets a preset similarity condition;

[0012] Determine whether the similar reference material element corresponding to the found source sentence is empty;

[0013] If the similar reference material element corresponding to the source sentence found out is not empty, determining whether any similar reference material element corresponding to the source sentence is contained in the source sentence;

[0014] If it is determined that any similar reference material element corresponding to the source sentence is contained in the source sentence, outputting the detection result of the source sentence;

[0015] If the similar reference material element corresponding to the source sentence is empty, or it is determined that the similar reference material elements corresponding to the source sentence are not contained in the source sentence, then for each sub-proposition corresponding to the source sentence, based on the keywords of the sub-proposition and the keywords of each reference material element in the reference material, multiple similar structure reference material elements are extracted;

[0016] Screening out a plurality of similar structure reference material elements having the greatest semantic similarity to the sub-proposition, and obtaining similar reference material elements corresponding to the sub-proposition;

[0017] Using each similar reference material element corresponding to the sub-proposition, perform implication judgment on the sub-proposition, and determine the detection result of the sub-proposition according to the implication judgment result;

[0018] The detection results of each of the source sentences and the detection results of each of the sub-propositions are statistically analyzed to determine the hallucination detection result of the large model.

[0019] Optionally, based on the above-mentioned large model hallucination detection method, the symbol segmentation of the large model output content is performed to obtain multiple source sentences, and the semantic segmentation of the large model output content is performed to obtain multiple sub-propositions, including:

[0020] Dividing the output content of the large model according to the sentence segmentation symbols to obtain multiple complete sentences;

[0021] Splitting each of the complete sentences according to commas to obtain a plurality of source sentences;

[0022] The large model is used to perform semantic segmentation on each of the complete sentences to obtain multiple sub-propositions.

[0023] Optionally, based on the above large model hallucination detection method, the step of extracting triplet knowledge from the file to be analyzed to obtain reference materials includes:

[0024] Preprocessing the file to be analyzed;

[0025] Based on the file to be analyzed and the task requirement information, multiple combinations of entity types and relationship types are extracted and stored in a dictionary;

[0026] Based on the combination of each entity type and relationship type in the dictionary, a plurality of entity-relationship triples are extracted from the preprocessed file to be analyzed using an information extraction model; wherein one entity-relationship triple includes a head entity, a relationship, and a tail entity;

[0027] Determine whether extraction based on grammatical rules is required according to the preset extraction rules;

[0028] If it is determined that extraction based on grammatical rules is not necessary, each of the entity-relationship triples is used as the reference material;

[0029] If it is determined that extraction needs to be performed based on grammatical rules, the preprocessed file to be analyzed is input into a grammar tool for grammatical analysis, and based on the grammatical analysis results, the subject, predicate and object in each sentence are extracted to form grammatical triples;

[0030] After weight reduction is performed on each of the entity-relationship triples and each of the grammatical triples, they are determined as the reference materials.

[0031] Optionally, in the above-mentioned large model hallucination detection method, before searching for a plurality of similar reference material elements corresponding to each source sentence from the reference material, the method further includes:

[0032] Replacing the proper nouns in each of the source sentences with corresponding simplified pronouns;

[0033] Each sentence in the reference material is segmented into long and short sentences, and the proper nouns in each segmented sentence are replaced with corresponding simplified pronouns to obtain reference material elements.

[0034] Optionally, in the above-mentioned large model hallucination detection method, before extracting a plurality of reference material elements with similar structures based on keywords of the sub-propositions and keywords of each reference material element in the reference material for each sub-proposition corresponding to the source sentence, the method further includes:

[0035] For each target sentence, respectively, the proper nouns in the target sentence are extracted by a named entity recognition model; wherein the target sentence includes each of the sub-propositions and each of the reference material elements;

[0036] Extracting fine-grained words from the target sentence based on grammatical rules;

[0037] Using the proper nouns in the target sentence as a dictionary, extracting keywords from the target sentence using an unsupervised keyword extraction method, and obtaining unsupervised keywords of the target sentence;

[0038] The proper nouns, fine-grained words and unsupervised keywords of the target sentence are combined to form keywords of the target sentence.

[0039] Optionally, in the above-mentioned large model hallucination detection method, before extracting multiple reference material elements of similar structure based on the keywords of the sub-proposition and the keywords of each reference material element in the reference material, the method further includes:

[0040] The pronouns in the sub-proposition are restored, and the proper nouns in the sub-proposition are replaced with simplified pronouns.

[0041] Optionally, based on the above-mentioned large model hallucination detection method, the detection results of each of the source sentences and the detection results of each of the sub-propositions are counted to determine the hallucination detection result of the large model, including:

[0042] The proportion of detection results indicating the presence of hallucinations in the detection results of each of the source sentences and the detection results of each of the sub-propositions is counted to obtain the output hallucination occurrence ratio of the large model.

[0043] The second aspect of the present application provides a large model hallucination detection device, comprising:

[0044] A segmentation unit, used to perform symbol segmentation on the output content of the large model to obtain multiple source sentences, and to perform semantic segmentation on the output content of the large model to obtain multiple sub-propositions;

[0045] A source tracing unit, used for tracing the source of each of the sub-propositions to obtain the sub-propositions corresponding to each of the source sentences;

[0046] A knowledge extraction unit is used to extract triplet knowledge from the document to be analyzed to obtain reference materials;

[0047] A first search unit is configured to search for a plurality of similar reference material elements corresponding to each source sentence from the reference material, respectively; wherein the similar reference material corresponding to the source sentence refers to a reference material element whose semantic similarity with the source sentence satisfies a preset similarity condition;

[0048] A first judging unit is used to judge whether the similar reference material element corresponding to the found source sentence is empty;

[0049] A second judgment unit is configured to judge whether any similar reference material element corresponding to the source sentence is contained in the source sentence when the similar reference material element corresponding to the source sentence found out is not empty;

[0050] A result output unit, configured to output a detection result of the source sentence when it is determined that any similar reference material element corresponding to the source sentence is contained in the source sentence;

[0051] A second search unit is configured to extract a plurality of similar structure reference material elements for each sub-proposition corresponding to the source sentence based on keywords of the sub-proposition and keywords of each reference material element in the reference material when the similar reference material element corresponding to the source sentence is empty or it is determined that the similar reference material elements corresponding to the source sentence are not contained in the source sentence;

[0052] A screening unit, used for screening out a plurality of similar structure reference material elements having the greatest semantic similarity with the sub-proposition, to obtain similar reference material elements corresponding to the sub-proposition;

[0053] A third judgment unit is used to perform implication judgment on the sub-proposition by using each similar reference material element corresponding to the sub-proposition, and determine the detection result of the sub-proposition according to the implication judgment result;

[0054] The result statistics unit is used to collect statistics on the detection results of each source sentence and the detection results of each sub-proposition to determine the hallucination detection result of the large model.

[0055] Optionally, in the above-mentioned large model hallucination detection device, the segmentation unit includes:

[0056] A coarse-grained segmentation unit, used to divide the output content of the large model according to sentence segmentation symbols to obtain multiple complete sentences;

[0057] A fine-grained segmentation unit, used to segment each of the complete sentences according to commas to obtain a plurality of source sentences;

[0058] The semantic segmentation unit is used to use the large model to perform semantic segmentation on each of the complete sentences to obtain multiple sub-propositions.

[0059] Optionally, in the above-mentioned large model hallucination detection device, the knowledge extraction unit includes:

[0060] A preprocessing unit, used for preprocessing the file to be analyzed;

[0061] A type determination unit, used to extract a plurality of entity types and relationship type combinations based on the to-be-analyzed file and task requirement information and store them in a dictionary;

[0062] An entity relationship extraction unit, configured to extract a plurality of entity-relationship triples from the preprocessed file to be analyzed based on the combination of each entity type and relationship type in the dictionary using an information extraction model; wherein one entity-relationship triple includes a head entity, a relationship, and a tail entity;

[0063] A fourth judgment unit, used to judge whether extraction based on grammatical rules is required according to preset extraction rules;

[0064] A first material determination unit, configured to use each of the entity-relationship triples as the reference material when it is determined that extraction based on grammatical rules is not necessary;

[0065] A grammar extraction unit is used for inputting the preprocessed file to be analyzed into a grammar tool for grammar analysis when it is determined that extraction based on grammar rules is required, and extracting the subject, predicate and object in each sentence according to the grammar analysis result to form a grammar triple;

[0066] The second material determination unit is used to reduce the weight of each of the entity-relationship triples and each of the grammatical triples and determine them as the reference material.

[0067] Optionally, in the above-mentioned large model hallucination detection device, it also includes:

[0068] A first replacement unit, used to replace the proper nouns in each of the source sentences with corresponding simplified pronouns;

[0069] The second replacement unit is used to segment each sentence in the reference material into long and short sentences, and replace the proper nouns in each segmented sentence with corresponding simplified pronouns to obtain reference material elements.

[0070] Optionally, in the above-mentioned large model hallucination detection device, it also includes:

[0071] A noun extraction unit, used for extracting proper nouns in each target sentence through a named entity recognition model; wherein the target sentence includes each of the sub-propositions and each of the reference material elements;

[0072] A fine-grained word extraction unit, used to extract fine-grained words in the target sentence based on grammatical rules;

[0073] An unsupervised keyword extraction unit, configured to extract keywords from the target sentence using the proper nouns in the target sentence as a dictionary and using an unsupervised keyword extraction method to obtain unsupervised keywords of the target sentence;

[0074] The keyword determination unit is used to combine the proper nouns, fine-grained words and unsupervised keywords of the target sentence into keywords of the target sentence.

[0075] Optionally, in the above-mentioned large model hallucination detection device, it also includes:

[0076] The third replacement unit is used to restore the pronouns in the sub-proposition and replace the proper nouns in the sub-proposition with simplified pronouns.

[0077] Optionally, in the above-mentioned large model hallucination detection device, the result statistics unit includes:

[0078] The result statistics subunit is used to count the detection results of each of the source sentences and the detection results of each of the sub-propositions, and the proportion of the detection results indicating the existence of hallucinations, so as to obtain the output hallucination occurrence ratio of the large model.

[0079] A third aspect of the present application provides an electronic device, including:

[0080] Memory and processor;

[0081] Wherein, the memory is used to store programs;

[0082] The processor is used to execute the program, and when the program is executed, it is specifically used to implement the large model hallucination detection method as described in any one of the above.

[0083] In a fourth aspect, the present application provides a computer storage medium for storing a computer program, wherein when the computer program is executed by a processor, the computer program is used to implement the large model hallucination detection method as described in any one of the above.

[0084] The present application provides a large model hallucination detection method, which performs symbol segmentation on the output content of the large model to obtain multiple source sentences, so as to take into account that Chinese expressions may contain more expressions. Through symbol segmentation, it is convenient to perform fine-grained analysis. In addition, the output content of the large model is semantically segmented to obtain multiple sub-propositions, so as to include that there may be multiple expressions of the same meaning. Through semantic analysis, its clear semantics can be obtained. Each sub-proposition is traced to the source to obtain the sub-proposition corresponding to each source sentence, which is convenient for the subsequent analysis of the source sentence in combination with the sub-proposition corresponding to the source sentence, not only limited to the analysis of the source sentence itself, but also considering its semantic situation. Then, the triple knowledge extraction is performed on the file to be analyzed to obtain the reference material. During the analysis, for each source sentence, multiple similar reference material elements corresponding to the source sentence are found from the reference material; wherein, the similar reference material corresponding to the source sentence refers to the reference material element whose semantic similarity with the source sentence meets the preset similarity condition, thereby obtaining the reference material that is most likely to support the source sentence. Then, it is determined whether the similar reference material element corresponding to the found source sentence is empty. If the similar reference material element corresponding to the source sentence found is not empty, then it is determined whether any similar reference material element corresponding to the source sentence is contained in the source sentence. If it is determined that any similar reference material element corresponding to the source sentence is contained in the source sentence, it means that the source sentence is supported by the reference material, so the detection result of the source sentence is output at this time. If the similar reference material element corresponding to the source sentence found is empty, or it is determined that each similar reference material element corresponding to the source sentence is not contained in the source sentence, it means that the source sentence is not supported by the reference material. However, considering that it may be a problem of its expression method, at this time, each sub-proposition corresponding to the source sentence will be further extracted based on the keywords of the sub-proposition and the keywords of each reference material element in the reference material, and then multiple similar structure reference material elements with the greatest semantic similarity to the sub-proposition will be screened out to obtain the similar reference material elements corresponding to the sub-proposition, so as to quickly find the reference material that is most likely to support the sub-proposition. Similarly, the sub-proposition is implied by using each similar reference material element corresponding to the sub-proposition, and the detection result of the sub-proposition is determined according to the result of the implied judgment. Finally, the detection results of each source sentence and each sub-proposition are statistically analyzed to determine the hallucination detection results of the large model. Thus, through symbol segmentation and semantic segmentation, fine-grained and semantic hallucination detection analysis of the output content is achieved, which is more in line with the characteristics of Chinese output content, and thus a large-model Chinese output content hallucination detection method that can effectively ensure the accuracy of the detection results is realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0086] Figure 1 A flowchart of a large model hallucination detection method provided in an embodiment of the present application;

[0087] Figure 2 A flowchart of a method for performing symbol segmentation and semantic segmentation on a large model output content provided in an embodiment of the present application;

[0088] Figure 3 A flowchart of a method for extracting triple knowledge from a file to be analyzed provided in an embodiment of the present application;

[0089] Figure 4 A flowchart of a keyword extraction method provided in an embodiment of the present application;

[0090] Figure 5 A schematic diagram of the architecture of a large-model hallucination detection device provided in an embodiment of the present application;

[0091] Figure 6 A schematic diagram of the architecture of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0092] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0093] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0094] The present application embodiment provides a large model hallucination detection method, such as Figure 1 As shown, the following steps are included:

[0095] S101. Perform symbol segmentation on the output content of the large model to obtain multiple source sentences, and perform semantic segmentation on the output content of the large model to obtain multiple sub-propositions.

[0096] It should be noted that for a longer Chinese content, it may usually express more meanings, and the content expressing different meanings is usually separated by punctuation marks. Therefore, in order to perform fine-grained analysis of different expressions, in an embodiment of the present application, the large model output content will first be segmented according to symbols to obtain multiple short sentences.

[0097] Since the existence of hallucinations is determined by implication judgment in the embodiment of the present application, and the same meaning can have multiple Chinese expressions, hallucination detection is performed solely from the output content of the large model, that is, detection is performed only from one expression of the large model output, which is easy to affect the accuracy of the detection result. Therefore, in the embodiment of the present application, the output content of the large model is also semantically segmented to analyze the real semantics as a sub-proposition of the original output content for subsequent hallucination detection. Since the short sentences segmented according to symbols belong to the original output content of the large model, and the content of semantic segmentation is obtained based on the analysis of these short sentences, the short sentences obtained by symbol segmentation are used as source sentences, and the sentences obtained by semantic segmentation are used as sub-propositions.

[0098] Optionally, in another embodiment of the present application, a specific implementation of step S101 is as follows: Figure 2 As shown, the following steps are included:

[0099] S201. Divide the output content of the large model according to sentence segmentation symbols to obtain multiple complete sentences.

[0100] It should be noted that since different sentences usually express different meanings, and different sentences are divided by short sentence symbols, such as periods, question marks, and exclamation marks, in an embodiment of the present application, the output content of the large model is first segmented according to coarse granularity, that is, the output content of the large model is divided according to sentence segmentation symbols to obtain multiple complete sentences.

[0101] S202, split each complete sentence according to the comma to obtain multiple source sentences.

[0102] It should be noted that the comma in a sentence is usually used to separate the more detailed expressions in the sentence. Therefore, in order to perform a more detailed analysis, in the embodiment of the present application, it will be further divided according to the comma.

[0103] S203. Use the large model to perform semantic segmentation on each complete sentence to obtain multiple sub-propositions.

[0104] Specifically, it can be a prompt based on the set speech segmentation, which converts a complete sentence into multiple word propositions from the semantic level through a large model.

[0105] S102. Trace the source of each sub-proposition to obtain the sub-proposition corresponding to each source sentence.

[0106] It should be noted that since the sentences obtained by semantic segmentation are the expressions of the corresponding source sentences, they should be processed as a set of data. Therefore, in order to determine the source sentences corresponding to each sub-proposition, it is necessary to trace the source of each sub-proposition so as to find the sub-propositions corresponding to each source sentence.

[0107] S103, extracting triple knowledge from the document to be analyzed to obtain reference materials.

[0108] It should be noted that in order to facilitate the accuracy of the comparison between the subsequent reference materials and the output content of the large model, in the embodiment of the present application, the original materials are no longer directly used as reference materials, but important information is extracted from them to form a reference material subset for hallucination analysis.

[0109] Therefore, in the embodiment of the present application, the associated information is analyzed from the file to be analyzed to form triplets as reference materials.

[0110] Optionally, in another embodiment of the present application, a specific implementation of step S103 is as follows: Figure 3 As shown, the following steps are included:

[0111] S301, pre-processing the file to be analyzed.

[0112] Specifically, the user can input the file to be analyzed in the form of text. In order to facilitate the subsequent information extraction, the file to be analyzed is first pre-processed by sentence segmentation, pronoun conversion, subject addition, etc.

[0113] S302: Based on the file to be analyzed and the task requirement information, multiple combinations of entity types and relationship types are extracted and stored in a dictionary.

[0114] It should be noted that in the embodiment of the present application, entity-relationship triple information extraction is required, so in order to determine which entities and corresponding relationships need to be extracted, it is necessary to first combine multiple entity types and relationship types based on the file to be analyzed and the task requirement information, that is, extract multiple entity type-relationship type information and store it in the Schema dictionary. For example, extract: {"name": "graduation school"}.

[0115] S303 . Based on the combination of each entity type and relationship type in the dictionary, a plurality of entity-relationship triples are extracted from the preprocessed file to be analyzed using an information extraction model.

[0116] Among them, an entity-relationship triple includes a head entity, a relationship, and a tail entity. For example, for the combination of {"name":"graduation school"}, the triple of (Xiaohong, graduation school, Peking University) is extracted. Among them, "Xiaohong" is the head entity, "graduation school" is the relationship, and "Peking University" is the tail entity.

[0117] S304: Determine whether extraction based on grammatical rules is required according to preset extraction rules.

[0118] It should be noted that, considering that the combination of the set entity type and relationship type is not comprehensive enough, the information extracted by the information extraction model may not be comprehensive enough. Therefore, in the embodiment of the present application, it is possible to further analyze whether it is necessary to further extract information through grammatical rules, such as whether the extraction result based on the model has a returned result, and whether the index value of the extracted entity-relationship in the sentence can cover the complete sentence, to determine whether it is necessary to extract based on grammatical rules.

[0119] If it is determined that the extraction is not required based on the grammatical rules, step S305 is executed. If it is determined that the extraction is required based on the grammatical rules, step S306 is executed.

[0120] S305. Use each entity-relationship triple as reference material.

[0121] S306: Input the preprocessed file to be analyzed into a grammar tool for grammar analysis, and extract the subject, predicate and object in each sentence according to the grammar analysis result to form a grammar triple.

[0122] Specifically, the subject, predicate and object in each sentence are analyzed from the document to be analyzed, and the relationship of the subject, predicate and object existing at the same time is retained.

[0123] S307, after reducing the weight of each entity-relationship triple and each grammatical triple, determine them as reference materials.

[0124] S104: for each source sentence, find out from the reference material a plurality of similar reference material elements corresponding to the source sentence.

[0125] The similar reference material corresponding to the source sentence refers to the reference material element whose semantic similarity with the source sentence satisfies a preset similarity condition. Optionally, the preset similarity condition may specifically be: the top N positions in the semantic similarity ranking with the source sentence, and the semantic similarity with the source sentence is greater than a preset similarity threshold.

[0126] Therefore, optionally, the semantic similarity between the source sentence and each reference material element in the reference material can be calculated, and then the reference material elements are sorted from large to small according to the semantic similarity. The reference material elements ranked in the top N positions are selected. Then, the reference material elements whose semantic similarity among the reference material elements ranked in the top N positions is determined as the similar reference material elements corresponding to the source sentence.

[0127] Optionally, since proper nouns may be long, which will affect the calculation of semantic similarity, in another embodiment of the present application, before executing step S104, the following may be further performed:

[0128] The proper nouns in each source sentence are replaced with corresponding simplified pronouns, and each sentence in the reference material is segmented into long and short sentences, and the proper nouns in each segmented sentence are replaced with corresponding simplified pronouns to obtain reference material elements.

[0129] For example, replace the full name of a company with a simple term such as "the company".

[0130] S105: Determine whether the similar reference material element corresponding to the source sentence found is empty.

[0131] Since the output content of the large model may be hallucinatory, it may not be supported by the reference material. Therefore, it may be impossible to find similar reference material elements corresponding to the source sentence, that is, there are no reference material elements that meet the preset similarity conditions.

[0132] If the similar reference material element corresponding to the source sentence is not empty, the source sentence can be further analyzed, so step S106 is executed at this time. If the similar reference material element corresponding to the source sentence is empty, it is necessary to consider the sub-proposition expressing its semantics for analysis, so step S108 is executed at this time.

[0133] S106, determining whether any similar reference material element corresponding to the source sentence is contained in the source sentence.

[0134] If it is determined that any similar reference material element corresponding to the source sentence is contained in the source sentence, it means that there is a reference material that supports the source sentence output by the large model, and there is no hallucination, so step S107 is executed at this time. If it is determined that none of the similar reference material elements corresponding to the source sentence is contained in the source sentence, it means that the reference material does not support the expression of the source sentence, so it is necessary to further analyze the sub-propositions expressing the same semantics.

[0135] S107: Output the detection result of the source sentence.

[0136] Optionally, the detection result of the source sentence may include similar reference material elements contained in the source sentence.

[0137] S108. For each sub-proposition corresponding to the source sentence, extract multiple reference material elements with similar structures based on the keywords of the sub-proposition and the keywords of each reference material element in the reference material.

[0138] Specifically, keywords of the sub-proposition and the reference material element are extracted, and then based on the matching degree of the keywords of the sub-proposition and the keywords of the reference material element, multiple reference material elements that are most similar in structure to the sub-proposition are extracted as similar structure reference material elements, so as to quickly obtain a large number of reference material elements that are most likely to be semantically similar to the sub-proposition, thereby facilitating faster search for reference material elements that are semantically similar to the sub-proposition. Optionally, the most similar multiple similar structure reference material elements can be screened out by calculating the overlap of keywords.

[0139] Optionally, in order to avoid the influence of longer proper nouns on similarity calculation, in another implementation of the present application, before executing step S108, the following may be further performed:

[0140] Restore the pronouns in the sub-proposition and replace the proper nouns in the sub-proposition with simplified pronouns.

[0141] Since some nouns are replaced by corresponding pronouns when the sub-proposition is generated, it is impossible to make an accurate replacement. Therefore, it is necessary to first restore the pronouns in the sub-proposition, and then replace the proper nouns in the sub-proposition with simplified pronouns.

[0142] Optionally, in another embodiment of the present application, a keyword extraction method is provided, such as Figure 4 As shown, the following steps are included:

[0143] S401 , for each target sentence, extract the proper nouns in the target sentence using a named entity recognition model.

[0144] The target sentence includes various sub-propositions and various reference material elements.

[0145] Since professional names are relatively special, they are usually the key words in a sentence, so they are extracted from the target sentence.

[0146] S402: Extract fine-grained words in the target sentence based on grammatical rules.

[0147] Since some keywords may also exist in words that are not proper nouns, in an embodiment of the present application, each word in a sentence is refined based on grammatical rules to obtain fine-grained words, and fine-grained words that may be keywords are extracted from them.

[0148] S403: Taking the proper nouns in the target sentence as a dictionary, extracting keywords from the target sentence using an unsupervised keyword extraction method to obtain unsupervised keywords of the target sentence.

[0149] Since proper nouns are sometimes referred to in more common terms, in order to extract keywords that are closer to the expected expression, an unsupervised keyword extraction method is used, using the extracted proper nouns as a dictionary for keyword extraction.

[0150] S404: The proper nouns, fine-grained words and unsupervised keywords of the target sentence are combined into keywords of the target sentence.

[0151] S109. Filter out multiple similar structure reference material elements with the greatest semantic similarity to the sub-proposition, and obtain similar reference material elements corresponding to the sub-proposition.

[0152] It should be noted that the extracted similar structural reference material elements are only similar in structure. In order to quickly reduce the number of extracted semantically similar elements, it is necessary to further screen out multiple similar structural reference material elements with the greatest semantic similarity to the sub-proposition from each similar structural reference material element from the perspective of semantic similarity as the similar reference material elements corresponding to the sub-proposition.

[0153] S110. Using each similar reference material element corresponding to the sub-proposition, perform implication judgment on the sub-proposition, and determine the detection result of the sub-proposition based on the implication judgment result.

[0154] Specifically, it is determined whether each similar reference material element corresponding to the sub-proposition is contained in the sub-proposition. If any similar reference material element corresponding to the sub-proposition is contained in the sub-proposition, it can be determined that the detection result of the sub-proposition is that there is no hallucination. Otherwise, it is determined that the detection result of the sub-proposition is that there is a sub-proposition.

[0155] S111. Count the detection results of each source sentence and each sub-proposition to determine the hallucination detection result of the large model.

[0156] Optionally, for source sentences whose detection results have not yet been determined, the detection results of their various sub-propositions can be statistically analyzed first, and the detection results of the source sentences can be determined based on the statistical results. Then, the detection results of each source sentence can be statistically analyzed to determine whether there are hallucinations in the output of the large model and the probability of hallucinations.

[0157] Optionally, in another embodiment of the present application, a specific implementation of step S111 includes:

[0158] The detection results of each source sentence and each sub-proposition are counted to find the proportion of detection results indicating the existence of hallucinations, and the proportion of output hallucinations in the large model is obtained.

[0159] The embodiment of the present application provides a large model hallucination detection method, which performs symbol segmentation on the output content of the large model to obtain multiple source sentences, so as to consider that Chinese expressions may contain more expressions. Through symbol segmentation, it is convenient to perform fine-grained analysis. In addition, the output content of the large model is semantically segmented to obtain multiple sub-propositions, so as to include that there may be multiple expressions of the same meaning. Through semantic analysis, its clear semantics can be obtained. Each sub-proposition is traced to the source to obtain the sub-proposition corresponding to each source sentence, which is convenient for the subsequent analysis of the source sentence in combination with the sub-proposition corresponding to the source sentence, not only limited to the analysis of the source sentence itself, but also considering its semantic situation. Then, the triple knowledge extraction is performed on the file to be analyzed to obtain the reference material. During the analysis, for each source sentence, multiple similar reference material elements corresponding to the source sentence are found from the reference material; wherein, the similar reference material corresponding to the source sentence refers to the reference material element whose semantic similarity with the source sentence meets the preset similarity condition, thereby obtaining the reference material that is most likely to support the source sentence. Then, it is determined whether the similar reference material element corresponding to the found source sentence is empty. If the similar reference material element corresponding to the source sentence found is not empty, then it is determined whether any similar reference material element corresponding to the source sentence is contained in the source sentence. If it is determined that any similar reference material element corresponding to the source sentence is contained in the source sentence, it means that the source sentence is supported by the reference material, so the detection result of the source sentence is output at this time. If the similar reference material element corresponding to the source sentence found is empty, or it is determined that each similar reference material element corresponding to the source sentence is not contained in the source sentence, it means that the source sentence is not supported by the reference material. However, considering that it may be a problem of its expression method, at this time, each sub-proposition corresponding to the source sentence will be further extracted based on the keywords of the sub-proposition and the keywords of each reference material element in the reference material, and then multiple similar structure reference material elements with the greatest semantic similarity to the sub-proposition will be screened out to obtain the similar reference material elements corresponding to the sub-proposition, so as to quickly find the reference material that is most likely to support the sub-proposition. Similarly, the sub-proposition is implied by using each similar reference material element corresponding to the sub-proposition, and the detection result of the sub-proposition is determined according to the result of the implied judgment. Finally, the detection results of each source sentence and each sub-proposition are statistically analyzed to determine the hallucination detection results of the large model. Thus, through symbol segmentation and semantic segmentation, fine-grained and semantic hallucination detection analysis of the output content is achieved, which is more in line with the characteristics of Chinese output content, and thus a large-model Chinese output content hallucination detection method that can effectively ensure the accuracy of the detection results is realized.

[0160] Another embodiment of the present application provides a large model hallucination detection device, such as Figure 5 As shown, it includes the following units:

[0161] The segmentation unit 501 is used to perform symbol segmentation on the output content of the large model to obtain multiple source sentences, and to perform semantic segmentation on the output content of the large model to obtain multiple sub-propositions.

[0162] The source tracing unit 502 is used to trace the source of each sub-proposition to obtain the sub-proposition corresponding to each source sentence.

[0163] The knowledge extraction unit 503 is used to extract triplet knowledge from the document to be analyzed to obtain reference materials.

[0164] The first search unit 504 is used to search for multiple similar reference material elements corresponding to each source sentence from the reference material, wherein the similar reference material corresponding to the source sentence refers to the reference material elements whose semantic similarity with the source sentence meets the preset similarity condition.

[0165] The first judging unit 505 is used to judge whether the similar reference material element corresponding to the found source sentence is empty.

[0166] The second judgment unit 506 is used to judge whether any similar reference material element corresponding to the source sentence is contained in the source sentence when the similar reference material element corresponding to the found source sentence is not empty.

[0167] The result output unit 507 is used to output the detection result of the source sentence when it is determined that any similar reference material element corresponding to the source sentence is contained in the source sentence.

[0168] The second search unit 508 is used to extract multiple similar structure reference material elements for each sub-proposition corresponding to the source sentence based on the keywords of the sub-proposition and the keywords of each reference material element in the reference material when the similar reference material element corresponding to the searched source sentence is empty, or when it is determined that the similar reference material elements corresponding to the source sentence are not contained in the source sentence.

[0169] The screening unit 509 is used to screen out multiple similar structure reference material elements with the greatest semantic similarity to the sub-proposition, and obtain similar reference material elements corresponding to the sub-proposition.

[0170] The third judgment unit 510 is used to perform implication judgment on the sub-proposition using the similar reference material elements corresponding to the sub-proposition, and determine the detection result of the sub-proposition according to the implication judgment result.

[0171] The result statistics unit 511 is used to collect statistics on the detection results of each source sentence and the detection results of each sub-proposition to determine the hallucination detection result of the large model.

[0172] Optionally, in a large model hallucination detection device provided in another embodiment of the present application, the segmentation unit includes:

[0173] The coarse-grained segmentation unit is used to divide the output content of the large model according to sentence segmentation symbols to obtain multiple complete sentences.

[0174] The fine-grained segmentation unit is used to segment each complete sentence according to the comma to obtain multiple source sentences.

[0175] The semantic segmentation unit is used to use the large model to perform semantic segmentation on each complete sentence to obtain multiple sub-propositions.

[0176] Optionally, in the large model hallucination detection device provided in another embodiment of the present application, the knowledge extraction unit includes:

[0177] The preprocessing unit is used to preprocess the files to be analyzed.

[0178] The type determination unit is used to extract a combination of multiple entity types and relationship types based on the file to be analyzed and the task requirement information and store them in the dictionary.

[0179] The entity-relationship extraction unit is used to extract multiple entity-relationship triples from the preprocessed files to be analyzed based on the combination of each entity type and relationship type in the dictionary using the information extraction model. An entity-relationship triple includes a head entity, a relationship, and a tail entity.

[0180] The fourth judgment unit is used to judge whether extraction based on grammatical rules is required according to preset extraction rules.

[0181] The first material determination unit is used to use each entity-relationship triple as a reference material when it is determined that extraction based on grammatical rules is not necessary.

[0182] The grammar extraction unit is used to input the preprocessed file to be analyzed into the grammar tool for grammar analysis when it is determined that extraction based on grammar rules is required, and according to the grammar analysis results, extract the subject, predicate and object in each sentence to form a grammar triple.

[0183] The second material determination unit is used to determine each entity-relationship triple and each grammatical triple as reference material after reducing their weight.

[0184] Optionally, in another embodiment of the present application, the large model hallucination detection device further includes:

[0185] The first replacement unit is used to replace the proper nouns in each source sentence with corresponding simplified pronouns.

[0186] The second replacement unit is used to segment each sentence in the reference material into long and short sentences, and replace the proper nouns in each segmented sentence with corresponding simplified pronouns to obtain reference material elements.

[0187] Optionally, in the above-mentioned large model hallucination detection device, it also includes:

[0188] The noun extraction unit is used to extract the proper nouns in each target sentence through the named entity recognition model, wherein the target sentence includes each sub-proposition and each reference material element.

[0189] The fine-grained word extraction unit is used to extract fine-grained words in the target sentence based on grammatical rules.

[0190] The unsupervised keyword extraction unit is used to use the proper nouns in the target sentence as a dictionary, and use the unsupervised keyword extraction method to extract keywords from the target sentence to obtain unsupervised keywords of the target sentence.

[0191] The keyword determination unit is used to combine the proper nouns, fine-grained words and unsupervised keywords of the target sentence into keywords of the target sentence.

[0192] Optionally, in another embodiment of the present application, the large model hallucination detection device further includes:

[0193] The third replacement unit is used to restore the pronouns in the sub-proposition and replace the proper nouns in the sub-proposition with simplified pronouns.

[0194] Optionally, in the large model hallucination detection device provided in another embodiment of the present application, the result statistics unit includes:

[0195] The result statistics sub-unit is used to count the detection results of each source sentence and each sub-proposition, and the proportion of the detection results indicating the existence of hallucinations, so as to obtain the output hallucination occurrence ratio of the large model.

[0196] It should be noted that the specific working process of each unit provided in the above embodiments of the present application can refer to the corresponding steps in the above method embodiments, and the steps are not repeated here.

[0197] Another embodiment of the present application provides an electronic device, such as Figure 6 As shown, including:

[0198] Memory 601 and processor 602 .

[0199] The memory 601 is used to store programs.

[0200] The processor 602 is used to execute the program stored in the memory 601. When the program is executed, it is specifically used to implement the large model hallucination detection method provided in any one of the above embodiments.

[0201] Another embodiment of the present application provides a computer storage medium for storing a computer program, which, when executed by a processor, is used to implement any of the large model hallucination detection methods described above.

[0202] Computer storage media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0203] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0204] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A large model hallucination detection method, characterized in that: include: Performing symbol segmentation on the output content of the large model to obtain multiple source sentences, and performing semantic segmentation on the output content of the large model to obtain multiple sub-propositions; Tracing the source of each of the sub-propositions to obtain the sub-propositions corresponding to each of the source sentences; Extract triplet knowledge from the document to be analyzed to obtain reference materials; For each of the source sentences, searching for multiple similar reference material elements corresponding to the source sentence from the reference material; wherein the similar reference material corresponding to the source sentence refers to a reference material element whose semantic similarity with the source sentence meets a preset similarity condition; Determine whether the similar reference material element corresponding to the found source sentence is empty; If the similar reference material element corresponding to the source sentence found out is not empty, determining whether any similar reference material element corresponding to the source sentence is contained in the source sentence; If it is determined that any similar reference material element corresponding to the source sentence is contained in the source sentence, outputting the detection result of the source sentence; If the similar reference material element corresponding to the source sentence is empty, or it is determined that the similar reference material elements corresponding to the source sentence are not contained in the source sentence, then for each sub-proposition corresponding to the source sentence, based on the keywords of the sub-proposition and the keywords of each reference material element in the reference material, multiple similar structure reference material elements are extracted; Screening out a plurality of similar structure reference material elements having the greatest semantic similarity to the sub-proposition, and obtaining similar reference material elements corresponding to the sub-proposition; Using each similar reference material element corresponding to the sub-proposition, perform implication judgment on the sub-proposition, and determine the detection result of the sub-proposition according to the implication judgment result; The detection results of each of the source sentences and the detection results of each of the sub-propositions are statistically analyzed to determine the hallucination detection result of the large model.

2. The method according to claim 1, characterized in that The symbol segmentation of the output content of the large model is performed to obtain multiple source sentences, and the semantic segmentation of the output content of the large model is performed to obtain multiple sub-propositions, including: Dividing the output content of the large model according to the sentence segmentation symbols to obtain multiple complete sentences; Splitting each of the complete sentences according to commas to obtain a plurality of source sentences; The large model is used to perform semantic segmentation on each of the complete sentences to obtain multiple sub-propositions.

3. The method according to claim 1, characterized in that The triplet knowledge extraction is performed on the document to be analyzed to obtain reference materials, including: Preprocessing the file to be analyzed; Based on the file to be analyzed and the task requirement information, multiple combinations of entity types and relationship types are extracted and stored in a dictionary; Based on the combination of each entity type and relationship type in the dictionary, a plurality of entity-relationship triples are extracted from the preprocessed file to be analyzed using an information extraction model; wherein one entity-relationship triple includes a head entity, a relationship, and a tail entity; Determine whether extraction based on grammatical rules is required according to the preset extraction rules; If it is determined that extraction based on grammatical rules is not necessary, each of the entity-relationship triples is used as the reference material; If it is determined that extraction needs to be performed based on grammatical rules, the preprocessed file to be analyzed is input into a grammar tool for grammatical analysis, and based on the grammatical analysis results, the subject, predicate and object in each sentence are extracted to form grammatical triples; After weight reduction is performed on each of the entity-relationship triples and each of the grammatical triples, they are determined as the reference materials.

4. The method according to claim 1, characterized in that: Before searching for a plurality of similar reference material elements corresponding to each source sentence from the reference material, the method further includes: Replacing the proper nouns in each of the source sentences with corresponding simplified pronouns; Each sentence in the reference material is segmented into long and short sentences, and the proper nouns in each segmented sentence are replaced with corresponding simplified pronouns to obtain reference material elements.

5. The method according to claim 1, characterized in that Before extracting a plurality of reference material elements with similar structures based on the keywords of the sub-propositions and the keywords of each reference material element in the reference material for each sub-proposition corresponding to the source sentence, the method further includes: For each target sentence, respectively, the proper nouns in the target sentence are extracted by a named entity recognition model; wherein the target sentence includes each of the sub-propositions and each of the reference material elements; Extracting fine-grained words from the target sentence based on grammatical rules; Using the proper nouns in the target sentence as a dictionary, extracting keywords from the target sentence using an unsupervised keyword extraction method, and obtaining unsupervised keywords of the target sentence; The proper nouns, fine-grained words and unsupervised keywords of the target sentence are combined to form keywords of the target sentence.

6. The method according to claim 1, characterized in that Before extracting a plurality of reference material elements with similar structures based on the keywords of the sub-proposition and the keywords of each reference material element in the reference material, the method further includes: The pronouns in the sub-proposition are restored, and the proper nouns in the sub-proposition are replaced with simplified pronouns.

7. The method according to claim 1, characterized in that The step of performing statistics on the detection results of each of the source sentences and the detection results of each of the sub-propositions to determine the hallucination detection result of the large model includes: The proportion of detection results indicating the presence of hallucinations in the detection results of each of the source sentences and the detection results of each of the sub-propositions is counted to obtain the output hallucination occurrence ratio of the large model.

8. A large model hallucination detection device, characterized in that: include: A segmentation unit, used to perform symbol segmentation on the output content of the large model to obtain multiple source sentences, and to perform semantic segmentation on the output content of the large model to obtain multiple sub-propositions; A source tracing unit, used for tracing the source of each of the sub-propositions to obtain the sub-propositions corresponding to each of the source sentences; A knowledge extraction unit is used to extract triplet knowledge from the document to be analyzed to obtain reference materials; A first search unit is configured to search for a plurality of similar reference material elements corresponding to each source sentence from the reference material, respectively; wherein the similar reference material corresponding to the source sentence refers to a reference material element whose semantic similarity with the source sentence satisfies a preset similarity condition; A first judging unit is used to judge whether the similar reference material element corresponding to the found source sentence is empty; A second judgment unit is configured to judge whether any similar reference material element corresponding to the source sentence is contained in the source sentence when the similar reference material element corresponding to the source sentence found out is not empty; A result output unit, configured to output a detection result of the source sentence when it is determined that any similar reference material element corresponding to the source sentence is contained in the source sentence; A second search unit is configured to extract a plurality of similar structure reference material elements for each sub-proposition corresponding to the source sentence based on keywords of the sub-proposition and keywords of each reference material element in the reference material when the similar reference material element corresponding to the source sentence is empty or it is determined that the similar reference material elements corresponding to the source sentence are not contained in the source sentence; A screening unit, used for screening out a plurality of similar structure reference material elements having the greatest semantic similarity with the sub-proposition, to obtain similar reference material elements corresponding to the sub-proposition; A third judgment unit is used to perform implication judgment on the sub-proposition by using each similar reference material element corresponding to the sub-proposition, and determine the detection result of the sub-proposition according to the implication judgment result; The result statistics unit is used to collect statistics on the detection results of each source sentence and the detection results of each sub-proposition to determine the hallucination detection result of the large model.

9. An electronic device, characterized in that: include: Memory and processor; Wherein, the memory is used to store programs; The processor is used to execute the program, and when the program is executed, it is specifically used to implement the large model hallucination detection method as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that: Used to store a computer program, which, when executed by a processor, is used to implement the large model hallucination detection method as described in any one of claims 1 to 7.

Images