Model illusion detection method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a model illusion detection method and device, electronic equipment and a storage medium, and belongs to the technical field of text processing and artificial intelligence. The method comprises the steps of obtaining an original text generated by a target model; performing named entity recognition on the original text to obtain a target entity phrase; position query is carried out on the target entity phrase to obtain a position probability, and the position probability represents the probability that any position in the original text is the target entity phrase; based on the position probability, performing illusion scoring on the original text to obtain a text illusion score; on the basis of the text illusion score and the original text, illusion detection is conducted on the target model, a model illusion detection result is obtained, and the model illusion detection result shows that illusion exists or does not exist in the text generated by the target model. According to the embodiment of the invention, the model illusion detection efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of text processing and artificial intelligence technology, and in particular to a model hallucination detection method and device, an electronic device and a storage medium. Background Art

[0002] Hallucination text refers to text whose content output by a large model does not match the actual text. Model hallucination detection refers to determining whether the text output by a large model is hallucination text. For example, in the medical field, by performing model hallucination detection on a large model that outputs diagnostic text that provides medical advice, the accuracy of the diagnostic text can be guaranteed.

[0003] At present, the model hallucination detection method usually inputs the same input text into a large model multiple times to obtain multiple output texts, and then performs consistency evaluation on the multiple output texts to determine whether the output text is a hallucination text. However, this method requires a lot of time and computing resources, resulting in low efficiency of model hallucination detection. Therefore, how to improve the efficiency of model hallucination detection has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a model hallucination detection method and device, an electronic device and a storage medium, aiming to improve the efficiency of model hallucination detection.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a model hallucination detection method, the method comprising:

[0006] Get the original text generated by the target model;

[0007] Performing named entity recognition on the original text to obtain a target entity phrase;

[0008] Performing a position query on the target entity phrase to obtain a position probability, wherein the position probability represents the probability of the target entity phrase being at any position in the original text;

[0009] Based on the position probability, performing hallucination scoring on the original text to obtain a text hallucination score;

[0010] Based on the text hallucination score and the original text, hallucination detection is performed on the target model to obtain a model hallucination detection result, wherein the model hallucination detection result indicates whether hallucination exists or not in the text generated by the target model.

[0011] In some embodiments, performing named entity recognition on the original text to obtain a target entity phrase includes:

[0012] Performing word segmentation processing on the original text to obtain text phrases;

[0013] Performing phrase recognition on the text phrase to obtain phrase information;

[0014] Based on the phrase information, a preset phrase part-of-speech set and a preset named entity type set, the text phrase is subjected to phrase screening to obtain the target entity phrase, wherein the phrase part-of-speech set is used to characterize the part-of-speech range of the target entity phrase, and the named entity type set is used to characterize the entity type range of the target entity phrase.

[0015] In some embodiments, the phrase information includes phrase part-of-speech information and phrase entity type, and the phrase screening of the text phrase based on the phrase information, a preset phrase part-of-speech set, and a preset named entity type set to obtain the target entity phrase includes:

[0016] Filtering out preliminary entity phrases whose phrase part-of-speech information is within the phrase part-of-speech set from the text phrases;

[0017] The target entity phrase whose phrase entity type is within the named entity type set is selected from the preliminary entity phrase.

[0018] In some embodiments, performing hallucination scoring on the original text based on the position probability to obtain a text hallucination score includes:

[0019] Based on the position probability and the target model, calculating the confidence of the target entity phrase to obtain a phrase hallucination score;

[0020] Based on a preset mapping table of phrase entity types and weight coefficients, the phrase hallucination scores are weighted summed to obtain the text hallucination score, wherein the mapping table of entity phrase types and weight coefficients is used to characterize the mapping relationship between the entity type and weight distribution of the target entity phrase.

[0021] In some embodiments, the calculating the confidence of the target entity phrase based on the position probability and the target model to obtain the phrase hallucination score includes:

[0022] Based on the target model, uncertainty calculation is performed on the target entity phrase to obtain phrase information entropy of the target entity phrase;

[0023] The phrase information entropy is quantified based on the position probability to obtain the phrase hallucination score.

[0024] In some embodiments, the weighted summing of the phrase hallucination scores based on a preset mapping table of phrase entity types and weight coefficients to obtain the text hallucination score includes:

[0025] Performing phrase type detection on the target entity phrase to obtain a first coefficient;

[0026] Determining a second coefficient of the target entity phrase based on the target entity phrase and a mapping table of the phrase entity type and the weight coefficient;

[0027] Combining the first coefficient and the second coefficient to obtain a target weight coefficient of the target entity phrase;

[0028] Based on the target weight coefficient, the phrase hallucination scores are weighted and summed to obtain the text hallucination score.

[0029] In some embodiments, performing hallucination detection on the target model based on the text hallucination score and the original text to obtain a model hallucination detection result includes:

[0030] Comparing the text hallucination score of the original text with a preset reference score;

[0031] If the text hallucination score of the original text is greater than or equal to the reference score, generating hallucination characterization data characterizing that the original text is a hallucination text, and performing data packaging on the hallucination characterization data to obtain the model hallucination detection result;

[0032] If the original text hallucination score is less than the reference score, non-hallucination representation data representing that the original text is a non-hallucination text is generated, and data packaging is performed on the non-hallucination representation data to obtain the model hallucination detection result.

[0033] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a model hallucination detection device, the device comprising:

[0034] The text acquisition module is used to obtain the original text generated by the target model;

[0035] An entity recognition module is used to perform named entity recognition on the original text to obtain a target entity phrase;

[0036] A probability query module, used to perform a position query on the target entity phrase to obtain a position probability, wherein the position probability represents the probability that any position in the original text is the target entity phrase;

[0037] A hallucination scoring module, used for performing hallucination scoring on the original text based on the position probability to obtain a text hallucination score;

[0038] The hallucination detection module is used to perform hallucination detection on the target model based on the text hallucination score and the original text to obtain a model hallucination detection result, wherein the model hallucination detection result indicates whether the text generated by the target model has hallucination or not.

[0039] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0040] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0041] The model hallucination detection method and device, electronic device and storage medium proposed in the present application can identify the target entity phrase by performing named entity recognition on the original text generated by the target model, and then perform position query on the target entity phrase to obtain the position probability, and perform hallucination scoring on the original text according to the position probability to obtain the text hallucination score. It can evaluate the model state of the original text generated by the target model in real time, obtain the information of the original text from the internal state of the target model, and improve the immediacy of model hallucination detection. In addition, by performing hallucination scoring on the target entity phrase, the calculation time of model hallucination detection can be reduced, thereby improving the efficiency of model hallucination detection. Furthermore, by quantifying the response uncertainty of the target model, the hallucination pattern of the potential target model can be revealed, which helps to improve the target model itself. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a flow chart of a model hallucination detection method provided in an embodiment of the present application;

[0043] Figure 2 yes Figure 1 Flow chart of step S102 in FIG.

[0044] Figure 3 yes Figure 2 Flow chart of step S203 in FIG.

[0045] Figure 4 yes Figure 1 Flow chart of step S104 in FIG.

[0046] Figure 5 yes Figure 4 Flow chart of step S401 in FIG.

[0047] Figure 6 yes Figure 4 Flow chart of step S402 in FIG.

[0048] Figure 7 yes Figure 1 Flow chart of step S105 in FIG.

[0049] Figure 8 is a schematic diagram of the structure of a model hallucination detection device provided in an embodiment of the present application;

[0050] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0054] First, some nouns involved in this application are analyzed:

[0055] Hallucination text: Hallucination text refers to information that is inconsistent with the facts or false in the natural language generation process. This phenomenon usually occurs when deep learning models generate content. The model may generate text that seems reasonable but is actually inaccurate based on patterns in the training data. The existence of hallucination text may lead to user misunderstanding or wrong decision-making. Therefore, identifying and detecting hallucination text has become an important task to improve the credibility of generated content. This false information can be effectively identified by analyzing the context, semantic consistency and entity recognition of the text.

[0056] Hallucination detection: Hallucination detection is a technique in natural language processing that aims to identify false or inaccurate information in generated text. In particular, when using deep learning models to generate text, the model sometimes produces content that is inconsistent with the facts, which is called "hallucination". Hallucination detection evaluates the authenticity and credibility of text by analyzing the context, semantic consistency, and entity recognition of the text, helping users identify and filter possible false information to improve the reliability and practicality of automatically generated content.

[0057] Hallucination text refers to text whose content output by a large model does not match the actual text. Model hallucination detection refers to determining whether the text output by a large model is hallucination text. For example, in the medical field, by performing model hallucination detection on a large model that outputs diagnostic text that provides medical advice, the accuracy of the diagnostic text can be guaranteed.

[0058] At present, the model hallucination detection method usually inputs the same input text into a large model multiple times to obtain multiple output texts, and then performs consistency evaluation on the multiple output texts to determine whether the output text is a hallucination text. However, this method requires a lot of time and computing resources, resulting in low efficiency of model hallucination detection. Therefore, how to improve the efficiency of model hallucination detection has become a technical problem that needs to be solved urgently.

[0059] Based on this, the embodiments of the present application provide a model hallucination detection method and device, an electronic device and a storage medium, aiming to improve the efficiency of model hallucination detection.

[0060] The model hallucination detection method and device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the model hallucination detection method in the embodiments of the present application is described.

[0061] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0062] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0063] The model hallucination detection method provided in the embodiment of the present application relates to the field of text processing and artificial intelligence technology. The model hallucination detection method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, or can be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the model hallucination detection method, etc., but is not limited to the above forms.

[0064] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0065] Figure 1 is an optional flow chart of the model hallucination detection method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S105.

[0066] Step S101, obtaining the original text generated by the target model;

[0067] Step S102, performing named entity recognition on the original text to obtain a target entity phrase;

[0068] Step S103, performing a position query on the target entity phrase to obtain a position probability, wherein the position probability represents the probability of any position in the original text being the target entity phrase;

[0069] Step S104, performing hallucination scoring on the original text based on the position probability to obtain a text hallucination score;

[0070] Step S105, based on the text hallucination score and the original text, hallucination detection is performed on the target model to obtain a model hallucination detection result, wherein the model hallucination detection result indicates whether the text generated by the target model has hallucination or not.

[0071] In steps S101 to S105 shown in the embodiment of the present application, the original text generated by the target model is obtained, and then named entity recognition is performed on the original text to obtain a target entity phrase. Secondly, a position query is performed on the target entity phrase to obtain the position probability of the target entity phrase at any position in the original text, and based on the position probability, the original text is hallucinated and scored to obtain a text hallucination score. Finally, based on the text hallucination score and the original text, hallucination detection is performed on the target model to obtain a model hallucination detection result indicating whether hallucinations exist or not in the text generated by the target model. Therefore, the present application can identify the target entity phrase by performing named entity recognition on the original text generated by the target model, and then perform a position query on the target entity phrase to obtain the position probability, and based on the position probability, perform hallucination scoring on the original text to obtain a text hallucination score. The application can evaluate the model state of the original text generated by the target model in real time, obtain information about the original text from the internal state of the target model, and improve the immediacy of model hallucination detection. In addition, by performing hallucination scoring on the target entity phrase, the calculation time of the model hallucination detection can be reduced, thereby improving the efficiency of the model hallucination detection. Furthermore, by quantifying the response uncertainty of the target model, the hallucination pattern of the potential target model can be revealed, which helps to improve the target model itself.

[0072] In step S101 of some embodiments, the target model may be a large model that outputs text according to input information, for example, a large language model, etc. The original text may be the text content output by the target model according to the input information.

[0073] In the embodiment of the present application, the original text output by the target model can be received by way of an interface call.

[0074] In step S102 of some embodiments, phrase information can be obtained by identifying information of each text phrase in the original text, and then text phrases that meet pre-set phrase information conditions are selected based on the phrase information to obtain the target entity phrase.

[0075] For details, see Figure 2 In some embodiments, step S102 may include but is not limited to steps S201 to S203:

[0076] Step S201, performing word segmentation processing on the original text to obtain text phrases;

[0077] Step S202, performing phrase recognition on the text phrase to obtain phrase information;

[0078] Step S203, based on the phrase information, the preset phrase part-of-speech set and the preset named entity type set, the text phrase is phrase-filtered to obtain the target entity phrase, wherein the phrase part-of-speech set is used to characterize the part-of-speech range of the target entity phrase, and the named entity type set is used to characterize the entity type range of the target entity phrase.

[0079] In step S201 of some embodiments, the original text is compared with a pre-prepared phrase library to achieve word segmentation processing of the original text, wherein the phrase library may be an open phrase library such as jieba, SpaCy, or NLTK.

[0080] In step S202 of some embodiments, the phrase information may be attribute information of the text phrase, wherein the attribute information may be phrase part-of-speech information, phrase entity type, and the like.

[0081] The present application can capture contextual information of text phrases, clarify the dependency relationship between each text phrase and other text phrases, and finally clarify the phrase information of each text phrase based on the dependency relationship of each text phrase.

[0082] In step S203 of some embodiments, the phrase part-of-speech set may be a set of phrases representing the part-of-speech of the target entity phrase, for example, the phrase part-of-speech set may include phrases representing the part-of-speech such as nouns and quantifiers. The named entity type set may be a set of phrases representing the type of the target entity phrase, for example, phrases representing the type of the target entity phrase such as names, dates, and organizations.

[0083] The present application can perform an initial screening of text phrases in the original text based on the phrase part-of-speech information in the phrase information to obtain preliminary entity phrases that conform to the phrase part-of-speech set, and then screen out entity phrases that conform to the named entity type set from the preliminary entity phrases based on the phrase entity type of the preliminary entity phrases to obtain the target entity phrases.

[0084] For details, see Figure 3 In some embodiments, step S203 may include but is not limited to steps S301 to S302:

[0085] Step S301, screening out preliminary entity phrases whose phrase part-of-speech information is within the phrase part-of-speech set from the text phrases;

[0086] Step S302 : Filter out target entity phrases whose phrase entity types are within the named entity type set from the preliminary entity phrases.

[0087] In step S301 of some embodiments, by comparing the phrase part-of-speech information of each text phrase with the phrase part-of-speech in the phrase part-of-speech set, it is determined whether each text phrase meets the phrase part-of-speech condition, so that preliminary entity phrases whose phrase part-of-speech information is within the phrase part-of-speech set can be screened out from the text phrases.

[0088] In step S302 of some embodiments, by comparing the phrase entity type of each preliminary entity phrase with the named entity types in the named entity type set, it is determined whether each preliminary entity phrase meets the conditions of the named entity type, so that the target entity phrase whose phrase entity type is within the named entity type set can be filtered out from the preliminary entity phrases.

[0089] In steps S301 to S302 shown in the present embodiment, preliminary screening of text phrases is achieved by screening out preliminary entity phrases whose phrase part-of-speech information is within a phrase part-of-speech set from text phrases, and then target entity phrases whose phrase entity types are within a named entity type set are screened out from the preliminary entity phrases to achieve secondary screening of text phrases, thereby improving the accuracy of the target entity phrases, avoiding the influence of non-important phrases on model hallucination detection, and thus improving the accuracy of model hallucination detection.

[0090] In steps S201 to S203 shown in the present embodiment, the original text is segmented to obtain text phrases contained in the original text, and then the text phrases are identified to obtain phrase information of the text phrases. Finally, the text phrases are screened according to the phrase information, a preset phrase part-of-speech set, and a preset named entity type set to obtain target entity phrases, which can ensure that the identified target entity phrases have important meanings in the original text, so as to improve the accuracy of model hallucination detection.

[0091] In step S103 of some embodiments, before the target model generates the original text, it is necessary to calculate the predicted text phrases that may appear at each position in the original text and the probability distribution of each predicted text phrase, and then select text phrases from the predicted text phrases according to the probability distribution to form the original text. For example, the predicted text phrases that may appear at the first position of the original text are "you, me, him, she", where the probability distribution of the phrase "you" is 10%, the probability distribution of the phrase "me" is 80%, the probability distribution of the phrase "he" is 5%, and the probability distribution of the phrase "she" is 5%. It can be determined that the text phrase at the first position in the original text is "me". Therefore, the present application can obtain the position probability of the target entity phrase by calling the confidence of the text phrase at each position calculated by the target model in the process of generating the original text.

[0092] In step S104 of some embodiments, the uncertainty of the target entity phrase is calculated based on the position probability of each target entity phrase, so as to obtain a phrase hallucination score for each target entity phrase, and then the phrase hallucination scores are weighted and summed to obtain a text hallucination score, wherein the uncertainty of the target entity phrase can be used to represent the probability that the original text contains the target entity phrase. For example, by retrieving the data of the original text generated by the target model, the context information related to the target entity phrase generated in the original text can be obtained. Furthermore, the amount of context information of the target entity phrase is calculated to obtain the uncertainty of the target entity phrase, thereby calculating the phrase hallucination score of the target entity phrase.

[0093] For details, see Figure 4 In some embodiments, step S104 may include but is not limited to steps S401 to S402:

[0094] Step S401, based on the position probability and the target model, the confidence of the target entity phrase is calculated to obtain a phrase hallucination score;

[0095] Step S402, based on a preset mapping table of phrase entity types and weight coefficients, weighted summation of phrase hallucination scores is performed to obtain a text hallucination score, wherein the mapping table of entity phrase types and weight coefficients is used to characterize the mapping relationship between entity types and weight distribution of target entity phrases.

[0096] In step S401 of some embodiments, the overall information entropy related to each target entity phrase may be calculated to obtain phrase information entropy, and then the phrase information entropy may be quantified to obtain a phrase hallucination score.

[0097] For details, see Figure 5 In some embodiments, step S401 may include but is not limited to steps S501 to S502:

[0098] Step S501, based on the target model, uncertainty calculation is performed on the target entity phrase to obtain the phrase information entropy of the target entity phrase;

[0099] Step S502: quantize the phrase information entropy based on the position probability to obtain a phrase hallucination score.

[0100] In step S501 of some embodiments, additional information of the target entity phrase can be obtained by calling the target model to generate contextual information related to the target entity phrase in the original text, and then the additional information is calculated to obtain the phrase information entropy representing the uncertainty of the target entity phrase.

[0101] In detail, the present application can calculate the phrase information entropy of the target entity phrase by the following formula:

[0102]

[0103] Among them, H represents the phrase information entropy, i represents the position of the target entity phrase in the original text, and H i represents the phrase information entropy of the i-th text phrase in the original text, v represents the target entity phrase, V represents the vocabulary generated by the target model in generating the original text, and p i (v) represents the probability distribution of each phrase in the vocabulary.

[0104] In step S502 of some embodiments, the position probability of the target entity phrase is logarithmically calculated to determine the degree of surprise of generating the target entity phrase, and then the degree of surprise of generating the target entity phrase is summed with the phrase information entropy of the target entity phrase to obtain a phrase hallucination score.

[0105] In detail, the present application can calculate the phrase hallucination score of the target entity phrase by the following formula:

[0106] h i = -log(p i (t i ))+H i

[0107] Where h represents the phrase hallucination score, h i represents the phrase hallucination score of the i-th text phrase in the original text, t i represents the i-th text phrase in the original text, p i (t i ) represents the position probability of the i-th text phrase in the original text.

[0108] In step S402 of some embodiments, the mapping table of phrase entity types and weight coefficients can represent the mapping relationship between the named entity types of the phrases and the weight ratios in the text hallucination score calculation. For example, the weight ratio of the named entity type "name" in the text hallucination score calculation can be 0.1, and the weight ratio of the named entity type "date" in the text hallucination score calculation can be 0.05.

[0109] By looking up a mapping table between phrase entity types and weight coefficients, the present application can determine the weight ratio of each target entity phrase in the text hallucination score calculation, and then perform weighted summation of the phrase hallucination scores of the target entity phrases based on the weight ratio to obtain the text hallucination score.

[0110] For details, see Figure 6In some embodiments, step S402 may include but is not limited to steps S601 to S604:

[0111] Step S601, performing phrase type detection on the target entity phrase to obtain a first coefficient;

[0112] Step S602, determining a second coefficient of the target entity phrase based on the target entity phrase and a mapping table of phrase entity types and weight coefficients;

[0113] Step S603, combining the first coefficient and the second coefficient to obtain a target weight coefficient of the target entity phrase;

[0114] Step S604: performing weighted sum processing on the phrase hallucination scores based on the target weight coefficient to obtain a text hallucination score.

[0115] In step S601 of some embodiments, a secondary detection is performed on the phrase entity type of the target entity phrase to determine whether the phrase entity type of the target entity phrase is within the above-mentioned named entity type set. If the phrase entity type of the target entity phrase is within the above-mentioned named entity type set, the first coefficient of the target entity phrase is 1; if the phrase entity type of the target entity phrase is not within the above-mentioned named entity type set, the first coefficient of the target entity phrase is 0.

[0116] In step S602 of some embodiments, the phrase entity type of the target entity phrase is obtained, and then the weight proportion corresponding to the phrase entity type in the mapping table of phrase entity types and weight coefficients is searched to determine the second coefficient of the target entity phrase.

[0117] In step S603 of some embodiments, the first coefficient and the second coefficient of each target entity phrase are multiplied and combined to determine the target weight coefficient of each target entity phrase.

[0118] In step S604 of some embodiments, a weighted summation process is performed based on the target weight coefficient and the phrase hallucination score of each target entity phrase, so as to calculate the text hallucination score of the original text, which is used to represent the uncertainty of the original text.

[0119] In steps S601 to S604 shown in the present embodiment, secondary type detection is performed on the target entity phrase to clarify the first coefficient of the target entity phrase, and then the second coefficient of the target entity phrase is determined based on a mapping table of the target entity phrase, phrase entity type and weight coefficient. Furthermore, a weighted summation process is performed on the phrase hallucination score of the target entity phrase based on the first coefficient and the second coefficient, and the text hallucination score of the original text can be obtained, thereby realizing hallucination detection of the original text to determine whether the original text contains erroneous information.

[0120] In steps S401 to S402 shown in the present embodiment, the confidence of the target entity phrase is calculated according to the position probability and the target model to obtain a phrase hallucination score, and then the phrase hallucination score is weighted and summed according to a preset mapping table of phrase entity types and weight coefficients to obtain a text hallucination score, which can reduce the impact of unnecessary entity phrases on the hallucination score of the original text and increase the impact of important entity phrases on the hallucination score of the original text, thereby improving the accuracy of the text hallucination score and the accuracy of the model hallucination detection.

[0121] In step S105 of some embodiments, after obtaining the text hallucination score of the original text, by comparing the text hallucination score with a preset reference score, it is possible to determine whether the original text output by the target model is a hallucination text.

[0122] For details, see Figure 7 In some embodiments, step S105 may include but is not limited to steps S701 to S703:

[0123] Step S701, comparing the text hallucination score of the original text with a preset reference score;

[0124] Step S702: if the text hallucination score of the original text is greater than or equal to the reference score, then generate hallucination characterization data that characterizes the original text as hallucination text, and perform data packaging on the hallucination characterization data to obtain a model hallucination detection result;

[0125] Step S703: If the original text hallucination score is less than the reference score, non-hallucination representation data representing that the original text is a non-hallucination text is generated, and the non-hallucination representation data is packaged to obtain a model hallucination detection result.

[0126] In step S701 of some embodiments, the reference score is pre-set data used to distinguish whether the original text is a hallucination text.

[0127] The present application can compare the text hallucination score of the original text with a preset reference score to achieve numerical comparison between the text hallucination score and the reference score.

[0128] In step S702 of some embodiments, when the text hallucination score of the original text is greater than or equal to the reference score, it means that there are more target entity phrases with high uncertainty in the original text, that is, the uncertainty of the original text is high. Therefore, the original text can be determined to be a hallucination text. At this time, hallucination representation data representing the original text as a hallucination text can be generated. Furthermore, the hallucination representation data is packaged to obtain a model hallucination detection result.

[0129] In step S703 of some embodiments, when the text hallucination score of the original text is less than the reference score, it means that there are fewer target entity phrases with high uncertainty in the original text, that is, the uncertainty of the original text is low. Therefore, the original text can be determined to be a non-hallucination text. At this time, non-hallucination representation data representing the original text as a non-hallucination text can be generated. Furthermore, the non-hallucination representation data is packaged to obtain a model hallucination detection result.

[0130] In steps S701 to S703 shown in the present embodiment, by comparing the text hallucination score of the original text with a preset reference score, and when the text hallucination score of the original text is greater than or equal to the reference score, the original text generated by the target model is determined to be a hallucination text, and when the original text hallucination score is less than the reference score, the original text generated by the target model is determined to be a non-hallucination text, and then based on the aforementioned determination information, a model hallucination detection result is generated, which can quickly and effectively realize hallucination detection of the target model, thereby improving the efficiency of model hallucination detection.

[0131] In an application scenario, when the target model is a medical diagnosis model and the original text is a medical diagnosis suggestion text, the output interface of the medical diagnosis model is called to obtain the medical diagnosis suggestion text output by the medical diagnosis model, and then the entity phrases in the medical diagnosis suggestion text are identified, such as drug names, symptoms and other phrases. Furthermore, the fully connected layer output interface in the medical diagnosis model is called to obtain the position probability of the entity phrase drug names and symptoms. Furthermore, based on the position probability of the entity phrase drug names and symptoms, each entity phrase is hallucinated and scored to obtain the phrase hallucination score of each entity phrase. Based on the phrase hallucination score, the text hallucination score of the medical diagnosis suggestion text is calculated. Finally, based on the text hallucination score, hallucination detection of the medical diagnosis model is implemented to generate a model hallucination detection result of the medical diagnosis model.

[0132] The present application performs named entity recognition on the original text generated by the target model to identify the target entity phrase, then performs a position query on the target entity phrase to obtain the position probability, and performs hallucination scoring on the original text based on the position probability to obtain a text hallucination score. The application can evaluate the model state of the original text generated by the target model in real time, obtain the information of the original text from the internal state of the target model, and improve the immediacy of model hallucination detection. In addition, by performing hallucination scoring on the target entity phrase, the calculation time of the model hallucination detection can be reduced, thereby improving the efficiency of the model hallucination detection. Furthermore, by quantifying the response uncertainty of the target model, the hallucination pattern of the potential target model can be revealed, which helps to improve the target model itself.

[0133] See also Figure 8The embodiment of the present application further provides a model hallucination detection device, which can implement the above-mentioned model hallucination detection method, and the device includes:

[0134] The text acquisition module 801 is used to acquire the original text generated by the target model;

[0135] An entity recognition module 802 is used to perform named entity recognition on the original text to obtain a target entity phrase;

[0136] The probability query module 803 is used to perform a position query on the target entity phrase to obtain a position probability, wherein the position probability represents the probability of the target entity phrase being at any position in the original text;

[0137] A hallucination scoring module 804 is used to perform hallucination scoring on the original text based on the position probability to obtain a text hallucination score;

[0138] The hallucination detection module 805 is used to perform hallucination detection on the target model based on the text hallucination score and the original text to obtain a model hallucination detection result, wherein the model hallucination detection result indicates whether the text generated by the target model has hallucination or not.

[0139] The specific implementation of the model hallucination detection device is basically the same as the specific implementation of the above-mentioned model hallucination detection method, and will not be repeated here.

[0140] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned model hallucination detection method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0141] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0142] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0143] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the model hallucination detection method of the embodiment of this application;

[0144] Input / output interface 903, used to implement information input and output;

[0145] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0146] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0147] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0148] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned model hallucination detection method is implemented.

[0149] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0150] The model hallucination detection method, model hallucination detection device, electronic device and storage medium provided in the embodiments of the present application obtain the original text generated by the target model, then perform named entity recognition on the original text to obtain the target entity phrase, then perform a position query on the target entity phrase to obtain the position probability of the target entity phrase at any position in the original text, and perform hallucination scoring on the original text based on the position probability to obtain a text hallucination score, and finally, perform hallucination detection on the target model based on the text hallucination score and the original text to obtain a model hallucination detection result indicating whether the text generated by the target model has hallucinations or not.

[0151] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0152] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0153] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0154] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0155] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0156] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0157] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0158] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0159] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0160] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0161] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A model hallucination detection method, characterized in that: The method comprises: Get the original text generated by the target model; Performing named entity recognition on the original text to obtain a target entity phrase; Performing a position query on the target entity phrase to obtain a position probability, wherein the position probability represents the probability of the target entity phrase being at any position in the original text; Based on the position probability, performing hallucination scoring on the original text to obtain a text hallucination score; Based on the text hallucination score and the original text, hallucination detection is performed on the target model to obtain a model hallucination detection result, wherein the model hallucination detection result indicates whether hallucination exists or not in the text generated by the target model.

2. The method according to claim 1, characterized in that The performing named entity recognition on the original text to obtain a target entity phrase includes: Performing word segmentation processing on the original text to obtain text phrases; Performing phrase recognition on the text phrase to obtain phrase information; Based on the phrase information, a preset phrase part-of-speech set and a preset named entity type set, the text phrase is subjected to phrase screening to obtain the target entity phrase, wherein the phrase part-of-speech set is used to characterize the part-of-speech range of the target entity phrase, and the named entity type set is used to characterize the entity type range of the target entity phrase.

3. The method according to claim 2, characterized in that The phrase information includes phrase part-of-speech information and phrase entity type, and the phrase screening of the text phrase based on the phrase information, a preset phrase part-of-speech set and a preset named entity type set to obtain the target entity phrase includes: Filtering out preliminary entity phrases whose phrase part-of-speech information is within the phrase part-of-speech set from the text phrases; The target entity phrase whose phrase entity type is within the named entity type set is selected from the preliminary entity phrase.

4. The method according to claim 1, characterized in that The step of performing hallucination scoring on the original text based on the position probability to obtain a text hallucination score includes: Based on the position probability and the target model, calculating the confidence of the target entity phrase to obtain a phrase hallucination score; Based on a preset mapping table of phrase entity types and weight coefficients, the phrase hallucination scores are weighted summed to obtain the text hallucination score, wherein the mapping table of entity phrase types and weight coefficients is used to characterize the mapping relationship between the entity type and weight distribution of the target entity phrase.

5. The method according to claim 4, characterized in that The confidence calculation of the target entity phrase based on the position probability and the target model to obtain a phrase hallucination score includes: Based on the target model, uncertainty calculation is performed on the target entity phrase to obtain phrase information entropy of the target entity phrase; The phrase information entropy is quantified based on the position probability to obtain the phrase hallucination score.

6. The method according to claim 4, characterized in that The method of performing weighted summation on the phrase hallucination scores based on a preset mapping table of phrase entity types and weight coefficients to obtain the text hallucination scores includes: Performing phrase type detection on the target entity phrase to obtain a first coefficient; Determining a second coefficient of the target entity phrase based on the target entity phrase and a mapping table of the phrase entity type and the weight coefficient; Combining the first coefficient and the second coefficient to obtain a target weight coefficient of the target entity phrase; Based on the target weight coefficient, the phrase hallucination scores are weighted and summed to obtain the text hallucination score.

7. The method according to any one of claims 1 to 6, characterized in that: The method of performing hallucination detection on the target model based on the text hallucination score and the original text to obtain a model hallucination detection result includes: Comparing the text hallucination score of the original text with a preset reference score; If the text hallucination score of the original text is greater than or equal to the reference score, generating hallucination characterization data characterizing that the original text is a hallucination text, and performing data packaging on the hallucination characterization data to obtain the model hallucination detection result; If the original text hallucination score is less than the reference score, non-hallucination representation data representing that the original text is a non-hallucination text is generated, and data packaging is performed on the non-hallucination representation data to obtain the model hallucination detection result.

8. A model hallucination detection device, characterized in that: The device comprises: The text acquisition module is used to obtain the original text generated by the target model; An entity recognition module is used to perform named entity recognition on the original text to obtain a target entity phrase; A probability query module, used to perform a position query on the target entity phrase to obtain a position probability, wherein the position probability represents the probability that any position in the original text is the target entity phrase; A hallucination scoring module, used for performing hallucination scoring on the original text based on the position probability to obtain a text hallucination score; The hallucination detection module is used to perform hallucination detection on the target model based on the text hallucination score and the original text to obtain a model hallucination detection result, wherein the model hallucination detection result indicates whether the text generated by the target model has hallucination or not.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the model hallucination detection method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the model hallucination detection method according to any one of claims 1 to 7 is implemented.

Images