Model illusion detection method and device, storage medium and electronic equipment

Abstract

The invention discloses a model illusion detection method and device, a storage medium and electronic equipment. The method comprises the steps that problem data are input into a target large model, a first output feature and a second output feature are determined, the first output feature represents a hidden state output feature of a middle layer of the target large model, and the second output feature represents an output feature obtained by splicing output features of M attention heads of the target large model; inputting the first output feature into a first classifier to obtain a first detection sub-result, and inputting the second output feature into a second classifier to obtain a second detection sub-result; under the condition that both the first detection sub-result and the second detection sub-result indicate that model illusion occurs in the target large model, an illusion prompt message is generated, and the illusion prompt message indicates that model parameters of the target large model need to be adjusted. The technical problem that the detection efficiency of model illusion is low is solved.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular, to a method and device for detecting model hallucinations, a storage medium, and an electronic device. Background Art

[0002] Existing hallucination detection methods, such as consistency-based detection, can identify some hallucination phenomena, but due to the need for multiple sampling and comparison, it is difficult for large language models to meet actual needs in terms of real-time and practicality. In other words, there is a technical problem in the related art that the detection efficiency of model hallucinations is low.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a method and device for detecting model hallucinations, a storage medium, and an electronic device, so as to at least solve the technical problem of low efficiency in detecting model hallucinations.

[0005] According to one aspect of an embodiment of the present application, a method for detecting model hallucination is provided, including: inputting problem data into a target large model, determining a first output feature and a second output feature, wherein the first output feature represents a hidden state output feature of an intermediate layer of the target large model, and the second output feature represents an output feature obtained by concatenating output features of M attention heads of the target large model, wherein the M attention heads are attention heads that meet a target output condition, and M is a positive integer; inputting the first output feature into a first classifier to obtain a first detection sub-result, and inputting the second output feature into a second classifier to obtain a second detection sub-result, wherein both the first detection sub-result and the second classifier are used to indicate whether model hallucination occurs in the target large model; generating a hallucination prompt message when both the first detection sub-result and the second detection sub-result indicate that model hallucination occurs in the target large model, wherein the hallucination prompt message indicates that model parameters of the target large model need to be adjusted.

[0006] According to another aspect of an embodiment of the present application, a device for detecting model hallucination is also provided, including: a determination module, used to input problem data into a target large model, determine a first output feature and a second output feature, wherein the first output feature represents a hidden state output feature of an intermediate layer of the target large model, and the second output feature represents an output feature obtained by concatenating output features of M attention heads of the target large model, wherein the M attention heads are attention heads that meet target output conditions, and M is a positive integer; a processing module, used to input the first output feature into a first classifier to obtain a first detection sub-result, and input the second output feature into a second classifier to obtain a second detection sub-result, wherein the first detection sub-result and the second classifier are both used to indicate whether model hallucination occurs in the target large model; a generation module, used to generate a hallucination prompt message when both the first detection sub-result and the second detection sub-result indicate that model hallucination occurs in the target large model, wherein the hallucination prompt message indicates that the model parameters of the target large model need to be adjusted.

[0007] Optionally, the device is also used to: generate a hallucination risk warning message when any one of the first detection sub-result and the second detection sub-result indicates that the target large model has the model hallucination, wherein the hallucination risk warning message indicates that it is necessary to detect the model parameters to determine whether the model parameters need to be adjusted; and generate answer data output by the target large model for the question data when both the first detection sub-result and the second detection sub-result indicate that the target large model has not the model hallucination.

[0008] Optionally, the device is also used for at least one of the following: setting the output hidden state parameter of the target large model to true to obtain the first output feature; obtaining the output feature of each of the M attention heads, and concatenating the output features to obtain the second output feature.

[0009] Optionally, the device is also used for: obtaining the output features of each of the M attention heads, and before concatenating the output features to obtain the second output features, inputting the training set data into the target large model to obtain the output features of each of the multiple attention heads; using the output features of each of the attention heads to respectively train a group of classifiers, wherein a classifier in the group of classifiers corresponds to one or more of the multiple attention heads; inputting the validation set data into the target large model to determine the classification accuracy parameters of each classifier in the group of classifiers; sorting the initial classification accuracy parameters of the group of classifiers according to the size of the values ​​to obtain a group of classification accuracy parameters; determining the classification accuracy parameters in the group of classification accuracy parameters whose values ​​are greater than or equal to the accuracy threshold as the target classification accuracy parameters; and determining the attention heads corresponding to the target classification accuracy parameters as the M attention heads.

[0010] Optionally, the device is also used to: obtain different types of seed question data; determine multiple initial question data from the seed question data in batches, and combine the multiple initial question data to obtain multiple sample question data; input the multiple sample question data into the target large model to obtain multiple groups of sample answer data, wherein one sample question data among the multiple sample question data corresponds to one group of sample answer data among the multiple groups of sample answer data; generate multiple sample question and answer pairs based on the multiple sample question data and the multiple groups of sample answer data, wherein one sample question and answer pair among the multiple sample question and answer pairs includes one sample question data, one sample answer data and a sample label, and the sample label reflects the association relationship between the one sample answer data and the one sample question data; use the multiple sample question and answer pairs to train the initial large model until the initial large model meets a preset convergence condition, and then determine the initial large model as the target large model.

[0011] Optionally, the device is used to generate multiple sample question and answer pairs based on the multiple sample question data and the multiple groups of sample answer data in the following manner: determine each sample question data in the multiple sample question data as the current sample question data in turn, and perform the following steps on the current sample question data to determine the multiple sample question and answer pairs: determine a group of current sample answer data corresponding to the current sample question data from the multiple groups of sample answer data; determine a basic sample answer data and multiple reference sample answer data from the group of current sample answer data; input the current sample question data and the group of current sample answer data into the target large model multiple times to obtain multiple reference sample labels corresponding to each reference sample answer data in the multiple reference sample answer data; based on the multiple reference sample labels, the current sample question data and the basic sample answer data generate a sample question and answer pair.

[0012] Optionally, the device is used to generate a sample question-answer pair based on the multiple reference sample labels, the current sample question data and the basic sample answer data in the following manner: respectively counting the number of occurrences of each sub-label in the multiple reference sample labels corresponding to each reference sample answer data, and determining the sub-label with the largest number of occurrences as the target reference label of the reference sample answer data corresponding thereto; determining the sub-label with the largest number of occurrences among the multiple target reference labels as the target basic label, wherein the target basic label corresponds to the basic sample answer data; and generating a sample question-answer pair based on the current sample question data, the basic sample answer data and the target basic label.

[0013] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned model hallucination detection method when running.

[0014] According to another aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned model hallucination detection method.

[0015] According to another aspect of the embodiments of the present application, there is also provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-mentioned model hallucination detection method through the computer program.

[0016] In an embodiment of the present application, the model hallucination is detected by inputting the problem data into the target large model and capturing and analyzing two types of output features at the same time, wherein the first output feature is the hidden state of the middle layer of the target large model, and the second output feature is formed by concatenating the output features of M specific attention heads that meet the target output conditions, where M is a positive integer representing the number of attention heads most relevant to the authenticity judgment within the model. The first output feature and the second output feature are respectively input into the pre-trained first classifier and the second classifier, and two independent detection sub-results can be obtained, which together indicate whether the target large model produces hallucinations that are inconsistent with the facts when processing specific inputs.

[0017] Furthermore, when both detection sub-results confirm that the model has hallucinations, a hallucination prompt message will be generated. This message not only identifies the existence of model hallucinations, but more importantly, the hallucination prompt message also provides guidance on adjusting model parameters to optimize the model's decision-making process and reduce the generation of hallucinations. By combining the hidden state of the intermediate layer with the output features of a specific attention head, the present invention achieves the purpose of more comprehensive and accurate detection of model hallucinations, realizes the real-time and high efficiency of the detection process, and thus solves the technical problem of low efficiency in model hallucination detection. This method not only improves the accuracy of detection, but also provides a basis for subsequent model parameter adjustments, enhancing the reliability of the model and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 is a schematic diagram of an application environment of an optional method for detecting model hallucination according to an embodiment of the present application;

[0020] Figure 2 is a flow chart of an optional method for detecting model hallucination according to an embodiment of the present application;

[0021] Figure 3 is a schematic diagram of an optional method for detecting model hallucination according to an embodiment of the present application;

[0022] Figure 4 is a schematic diagram of another optional method for detecting model hallucination according to an embodiment of the present application;

[0023] Figure 5 is a schematic diagram of another optional method for detecting model hallucination according to an embodiment of the present application;

[0024] Figure 6is a schematic diagram of another optional method for detecting model hallucination according to an embodiment of the present application;

[0025] Figure 7 is a schematic structural diagram of an optional model hallucination detection device according to an embodiment of the present application;

[0026] Figure 8 is a schematic structural diagram of an optional model hallucination detection product according to an embodiment of the present application;

[0027] Fig. 9 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] First, the rankings appearing in the examples of this application are explained:

[0031] Big language model: A big language model (big model) is a natural language processing model that uses deep learning technology and is trained on a large amount of text data. It is designed to understand, generate, and process natural language text. Today's popular big language models are based on the Transformer architecture, which is composed of multiple Transformer decoding blocks stacked together, and each Transformer decoding block contains network structures such as attention heads and feedforward layers. The big language model can use the richness of language to abstract concepts, sort out thought processes, understand complex queries, and formulate strategies and solutions in decision-making scenarios, providing a solid technical foundation for automation and intelligence.

[0032] Large model reasoning process: The large model reasoning process is the process of using a pre-trained large model to process new input data to generate prediction results. After the data is input into the large model, it will pass through the network structure of each layer of the large model. In models such as the Transformer architecture, complex feature extraction and transformation will be performed through modules such as the multi-head attention mechanism to continuously mine the potential semantics and structural information of the data. After these features are calculated and fused layer by layer, the prediction results are finally generated through the output layer.

[0033] Internal state of a large model: The internal state of a large model refers to the output values ​​of its intermediate sublayers during the reasoning process of the large language model. Although the outputs of these intermediate layers are difficult to be directly converted into natural language, they also contain rich semantic information. Studying the internal features of a large model helps humans understand the working process of the large model and enhance its interpretability.

[0034] Large model hallucination: The phenomenon that large language models generate content inconsistent with real-world facts or user input. This phenomenon can be divided into two categories: factual hallucination and fidelity hallucination. Factual hallucination involves the inconsistency between the content generated by the model and verifiable real-world facts, such as fabricated facts or factual inconsistency. Fidelity hallucination refers to the inconsistency between the generated content and user instructions or context, including instruction inconsistency, context inconsistency and logical inconsistency. The emergence of large model hallucination is usually closely related to the data source, training process and reasoning process. The data set may contain false information, bias or outdated knowledge, causing the model to learn inaccurate information during training. In addition, the model may produce hallucinations due to the difference between internal encoding and external behavior when processing input information and performing internal reasoning. The impact of hallucinations is multifaceted, and it may undermine users' trust in model output, especially in application areas such as the financial industry that require high accuracy.

[0035] The present application is described below in conjunction with embodiments:

[0036] According to one aspect of an embodiment of the present application, a method for detecting model hallucination is provided. Optionally, in this embodiment, the above-mentioned method for detecting model hallucination can be applied to Figure 1 In the hardware environment composed of the server 101 and the terminal device 103 shown in FIG. Figure 1 As shown, the server 101 is connected to the terminal device 103 via a network, and can be used to provide services for the terminal device or an application 107 installed on the terminal device. The application can be a video application, an instant messaging application, a browser application, an educational application, a game application, etc. A database 105 may be set up on the server or independently of the server to provide data storage services for the server 101, for example, a game data storage server. The above-mentioned network may include, but is not limited to, a wired network and a wireless network, wherein the wired network includes, a local area network, a metropolitan area network and a wide area network; the wireless network includes, Bluetooth, WIFI and other networks that implement wireless communication; the terminal device 103 may be a terminal configured with an application, and may include, but is not limited to, at least one of the following: a mobile phone (such as an Android phone, an iOS phone, etc.), a laptop computer, a tablet computer, a PDA, a MID (Mobile Internet Devices), a PAD, a desktop computer, a smart TV, an intelligent voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, a virtual reality (VR) terminal, an augmented reality (AR) terminal, a mixed reality (MR) terminal and other computer devices; the above-mentioned server may be a single server, a server cluster consisting of multiple servers, or a cloud server.

[0037] Combination Figure 1 As shown, the above-mentioned model hallucination detection method can be executed by an electronic device, and the electronic device can be a terminal device or a server. The above-mentioned model hallucination detection method can be implemented by the terminal device or the server respectively, or by the terminal device and the server together.

[0038] The above is only an example and is not specifically limited in this embodiment.

[0039] Optionally, as an optional implementation, as Figure 2 As shown, the detection method of the above model hallucination includes:

[0040] S202, inputting the problem data into the target large model, determining a first output feature and a second output feature, wherein the first output feature represents a hidden state output feature of the middle layer of the target large model, and the second output feature represents an output feature obtained by concatenating the output features of M attention heads of the target large model, where the M attention heads are attention heads that meet the target output condition, and M is a positive integer;

[0041] Optionally, in an embodiment of the present application, the above-mentioned target large model can be any large language model based on the Transformer architecture, including but not limited to the Qwen2-7B-Instruct model.

[0042] It should be noted that the method of obtaining the above-mentioned first output feature and the second output feature may vary depending on the specific model structure. For example, the hidden state of the middle layer may be in different positions in different model architectures, and the selection of M attention heads may also be based on different standards and methods. This application does not limit this.

[0043] In addition, the specific value of M can be adjusted according to application requirements and model performance. As long as these attention heads can effectively reflect the internal state of the model when processing factual questions and answers, they can be used in the hallucination detection process of the present invention.

[0044] It should also be noted that the classifier model structure used in the embodiments of the present application, such as a multilayer perceptron, can also select other suitable machine learning models according to actual conditions, such as a support vector machine or a neural network. As long as the classification purpose can be achieved, the present application does not limit this.

[0045] S204, inputting the first output feature into a first classifier to obtain a first detection sub-result, and inputting the second output feature into a second classifier to obtain a second detection sub-result, wherein both the first detection sub-result and the second classifier are used to indicate whether the target large model has model hallucination;

[0046] Optionally, in an embodiment of the present application, the first classifier can be any type of machine learning model, including but not limited to a multilayer perceptron (MLP), or a support vector machine (SVM), a decision tree, or a more complex deep learning model, as long as the model can accurately detect hallucinations based on the first output feature.

[0047] At the same time, the second output feature will be input into the second classifier to obtain the second detection sub-result. The second output feature is composed of the output features of M specific attention heads in the target large model, which are considered to be crucial for authenticity judgment. The second classifier is also trained to analyze the second output feature and further evaluate the risk of hallucination of the target large model during the answering process.

[0048] It should be noted that the training data, model structure and parameter settings of the first classifier and the second classifier can be adjusted according to different application scenarios and characteristics of the target large model.

[0049] For example, the construction of training datasets can be based on factual question answering in different fields, while the structure and parameters of the classifier can be optimized according to the type of model hallucination and detection requirements.

[0050] In addition, the first classifier and the second classifier can adopt different training strategies and optimization algorithms in practical applications to improve the accuracy and efficiency of detection. This application does not make specific limitations on this. As long as the classifier can effectively detect model hallucinations, its specific implementation details can be flexibly selected according to actual needs.

[0051] S206 , when both the first detection sub-result and the second detection sub-result indicate that model hallucination occurs in the target large model, generate a hallucination prompt message, wherein the hallucination prompt message indicates that model parameters of the target large model need to be adjusted.

[0052] Optionally, in an embodiment of the present application, the hallucination prompt message refers to the system providing the user with specific suggestions on model parameter adjustment based on the consensus of the first detection sub-result and the second detection sub-result, including but not limited to lowering the generation temperature of the model to reduce randomly constructed answers, adjusting the weights of the attention heads, optimizing the training data set of the model to include more positive examples of factual questions and answers, or increasing negative samples related to hallucination detection to help the model better learn the correct pattern of factual questions and answers during the training process.

[0053] It should be noted that the specific method of adjusting model parameters may vary depending on the architecture of the target large model and the specific application scenario. For example, for some models, this can be achieved by fine-tuning the hyperparameters of the training phase, such as learning rate, batch size, or optimization algorithm; while for other models, it may be necessary to adjust the model architecture itself more deeply, such as increasing or decreasing the number of layers, adjusting the parameters of the attention mechanism, etc. In addition, users or model developers can also adopt additional strategies based on hallucination prompt messages, such as increasing the proportion of supervised learning of the model, introducing more complex loss functions, or adopting more advanced training techniques, such as a combination of pre-training and fine-tuning. This application is not limited to this.

[0054] In an exemplary embodiment, assuming that the target macro model is Qwen2-7B-Instruct, a specific factual question (question data): "Which mountain is the highest peak in China?" is used as an example to demonstrate the entire hallucination detection process:

[0055] First, the question data is input into the target large model Qwen2-7B-Instruct, and the model starts its internal reasoning process to process the question and generate an answer. In this process, the hidden state output features of the middle layer of the model are recorded. These features constitute the first output features, reflecting the internal state and information encoding of the model when processing the question.

[0056] At the same time, the output features of the M specific attention heads of the model are considered to be most relevant to the authenticity judgment of the answers generated by the model. In this example, assuming that M = 28, that is, 28 attention heads are selected, the output features of these attention heads are concatenated to obtain the second output feature, which integrates the attention allocation information of the model when processing the input.

[0057] Next, the first output feature is input into the first classifier, which is trained to analyze whether the model is hallucinating based on the hidden state features. Similarly, the second output feature is also input into the second classifier, which is specifically used to analyze the output features of the attention head and judge the fidelity of the model's answer.

[0058] If both the first classifier and the second classifier conclude in this test that there is an illusion in the model's output, the model may incorrectly answer that the highest peak in China is not Mount Everest.

[0059] At this point, a hallucination prompt message will be generated, which will contain suggestions on how to adjust the parameters of the target large model. For example, you can adjust the model's training data set, add more geography-related factual question and answer examples, or fine-tune the model's attention mechanism parameters to improve the model's performance when processing factual questions. Through this detection and feedback mechanism, the model's performance is continuously optimized, improving its reliability in factual question and answer tasks.

[0060] That is, based on the internal state of the target large model, the output of the model's factual questions and answers is detected for hallucinations. In the current field of artificial intelligence, although large models have demonstrated powerful capabilities in natural language processing and question-answering systems, they often speak nonsense in a serious manner. This is the large model hallucination phenomenon. This illusion manifests itself in giving inaccurate, wrong or even fictitious answers, which seriously affects the application reliability and user experience of the large model. In combination with the embodiments of the present application, it is possible to quickly detect whether the output answer of the model has hallucinations. The model hallucination detection method proposed in the embodiments of the present application is integrated into the use of the target large model. When hallucinations appear in the answer of the target large model, the user can be reminded in time to prevent the user from being misled by the hallucinations output by the model, thereby improving the user's experience and the reliability of the large model.

[0061] In an embodiment of the present application, the model hallucination is detected by inputting the problem data into the target large model and capturing and analyzing two types of output features at the same time, wherein the first output feature is the hidden state of the middle layer of the target large model, and the second output feature is formed by concatenating the output features of M specific attention heads that meet the target output conditions, where M is a positive integer representing the number of attention heads most relevant to the authenticity judgment within the model. The first output feature and the second output feature are respectively input into the pre-trained first classifier and the second classifier, and two independent detection sub-results can be obtained, which together indicate whether the target large model produces hallucinations that are inconsistent with the facts when processing specific inputs.

[0062] Furthermore, when both detection sub-results confirm that the model has hallucinations, the system will generate a hallucination prompt message. This message not only identifies the existence of model hallucinations, but more importantly, the hallucination prompt message also provides guidance on adjusting model parameters to optimize the model's decision-making process and reduce the generation of hallucinations. By combining the hidden state of the intermediate layer with the output features of a specific attention head, the present invention achieves the purpose of more comprehensive and accurate detection of model hallucinations, realizes the real-time and high efficiency of the detection process, and thus solves the technical problem of low efficiency in model hallucination detection. This method not only improves the accuracy of detection, but also provides a basis for subsequent model parameter adjustments, enhancing the reliability of the model and user experience.

[0063] As an optional scheme, the above method also includes: when any one of the above-mentioned first detection sub-result and the above-mentioned second detection sub-result indicates that the above-mentioned target large model has the above-mentioned model hallucination, generating a hallucination risk warning message, wherein the above-mentioned hallucination risk warning message indicates that it is necessary to detect the above-mentioned model parameters to determine whether the above-mentioned model parameters need to be adjusted; when the above-mentioned first detection sub-result and the above-mentioned second detection sub-result both indicate that the above-mentioned target large model does not have the above-mentioned model hallucination, generating the answer data output by the above-mentioned target large model for the above-mentioned question data.

[0064] Optionally, in an embodiment of the present application, the first detection sub-result and the second detection sub-result refer to the judgments made by the first classifier and the second classifier based on the first output feature and the second output feature, respectively, which are used to indicate whether the target large model has model hallucinations when processing the problem data. The first output feature represents the hidden state output feature of the middle layer of the target large model, and the second output feature is the feature obtained by splicing the output features of M specific attention heads in the target large model. The judgment results of these classifiers provide a basis for subsequent hallucination detection and user feedback, including but not limited to generating hallucination prompt messages or hallucination risk prompt messages, and directly outputting model answers in the absence of hallucinations.

[0065] It should be noted that the generated answer data, hallucination risk warning message, and processing logic of the hallucination warning message can be adjusted according to the actual application scenario and user needs.

[0066] For example, in some scenarios, users want the system to automatically make preliminary adjustments or optimization suggestions to model parameters when hallucination risk is detected, rather than just providing risk warning messages; in other scenarios, users prefer to receive detailed hallucination risk analysis reports to facilitate manual review and subsequent model improvement work. This application does not limit this.

[0067] In an exemplary embodiment, when the target large model processes the question: "Which mountain is the highest in China?", the first classifier analyzes the first output feature, that is, the hidden state of the middle layer, and determines that the model's answer may deviate from the facts; at the same time, the second classifier also believes that the model's answer is at risk of not being faithful to the question based on the second output feature, that is, the output of M attention heads. In this case, even if only one classifier detects a hallucination, the system will generate a hallucination risk prompt message, prompting the user to check the model parameters to determine whether further adjustments are needed. This may include but is not limited to reviewing the model's hyperparameters, such as learning rate, temperature, attention head weights, etc., to ensure the performance and fidelity of the model on factual question-answering tasks.

[0068] Through the embodiments of the present application, a comprehensive analysis of the first detection sub-result and the second detection sub-result is adopted to achieve the technical effect of comprehensively evaluating the hallucination phenomenon of the target large model, thereby achieving the purpose of enhancing the user's trust in the output content of the large model while improving the reliability of the model, thereby ensuring the stability and accuracy of the model in automated and intelligent applications.

[0069] As an optional scheme, the above method also includes at least one of the following: setting the output hidden state parameter of the above target large model to true to obtain the above first output feature; obtaining the output feature of each of the above M attention heads, and splicing the above output features to obtain the above second output feature.

[0070] Optionally, in an embodiment of the present application, the output hidden state parameter of the target large model is true, which means that when the target large model processes problem data, the model parameter output_hidden_states is set to True to ensure that the model outputs the hidden state of each layer during the reasoning process, and can directly access and utilize the internal representation of the middle layer of the model, providing a basis for the subsequent first output feature extraction. The first output features refer to these hidden state output features, which reflect the internal state changes and information processing process of the model when processing specific problems, including but not limited to the activation state and feature representation of the middle layer.

[0071] It should be noted that the process of obtaining the output features of M attention heads and concatenating these features to obtain the second output features may vary due to different model architectures and implementation details. For example, in some models, the output features of the attention heads can be directly used as part of the output of the model; while in other models, additional layers or operations may be required to extract these features. In addition, the numerical selection of M can be adjusted according to the scale and complexity of the model, as well as the sensitivity requirements for hallucination detection, which is not limited in this application.

[0072] In an exemplary embodiment, assuming that the target large model is being used to process a series of factual questions, in order to obtain the first output feature, the output_hidden_states parameter is set to True, so that the model will output the hidden state of the intermediate layer while generating the answer. Next, in order to obtain the second output feature, the M attention heads that are most relevant to the authenticity of the model are first determined, and then the output features of each attention head are extracted respectively. These features are concatenated to form a comprehensive feature vector as the second output feature. These operations provide comprehensive model internal state information for subsequent hallucination detection, making the detection process more accurate and robust.

[0073] Through the embodiments of the present application, a method of obtaining a first output feature by outputting a hidden state parameter as true, and obtaining a second output feature by splicing the output features of M attention heads is adopted, thereby achieving a deep insight into the internal state of the target large model, and thereby effectively detecting the technical effect of model hallucinations, thereby achieving the purpose of improving the accuracy and reliability of the model output and enhancing the user experience.

[0074] As an optional scheme, before obtaining the output features of each of the M attention heads and concatenating the output features to obtain the second output features, the method further includes: inputting the training set data into the target large model to obtain the output features of each of the multiple attention heads; using the output features of each of the attention heads to train a group of classifiers respectively, wherein one classifier in the group of classifiers corresponds to one or more of the multiple attention heads; inputting the validation set data into the target large model to determine the classification accuracy parameters of each of the group of classifiers; sorting the initial classification accuracy parameters of the group of classifiers according to their values ​​to obtain a group of classification accuracy parameters; determining the classification accuracy parameters in the group of classification accuracy parameters whose values ​​are greater than or equal to the accuracy threshold as the target classification accuracy parameters; and determining the attention heads corresponding to the target classification accuracy parameters as the M attention heads.

[0075] Optionally, in an embodiment of the present application, the training set data refers to labeled question-answer pair data used to train a classifier, which includes question data and its corresponding multiple model answers, as well as labels obtained based on consistency review, i.e., whether the answer is hallucinating. The output features of each attention head obtained when the target large model processes the training set data refer to the intermediate representation generated by the attention mechanism of the model during the reasoning process, which can reflect the model's attention level and processing method for the input data. Each classifier in a set of classifiers corresponds to one of multiple attention heads, and is used to determine whether the model answer is hallucinating based on the output features of a specific attention head.

[0076] It should be noted that the classification accuracy parameters can be calculated and sorted in a variety of ways.

[0077] For example, when calculating the classification accuracy, it can be based on the validation set data, which usually contains a certain number of unseen question-answer pairs for evaluating the generalization ability of the classifier. The setting of the accuracy threshold can be flexibly adjusted according to the specific application scenario and the requirements for hallucination detection accuracy, and this application does not limit this.

[0078] In addition, when determining the target classification accuracy parameters and the corresponding M attention heads, factors such as the stability of the classifier and computational efficiency can be considered to ensure that the final selected attention heads and classifiers can perform the hallucination detection task efficiently and accurately.

[0079] In an exemplary embodiment, the training set data is first input into the target large model to obtain the output features related to different attention heads. Then, a group of classifiers are trained using the output features of each attention head. During the training process, the Adam optimizer is used to update the parameters, and the cross entropy loss function is used to measure the difference between the classifier's prediction and the actual label. The validation set data is input into the target large model, and the validation set data is classified by the classifier, and the classification accuracy parameters of each classifier are obtained by statistics. Sort by the size of the classification accuracy parameters, select the classifiers whose accuracy parameters meet the set accuracy threshold, and their corresponding attention heads are M attention heads.

[0080] Through the embodiments of the present application, the classifier training and attention head selection strategies are adopted to improve the accuracy of hallucination detection of the target large model, and can effectively identify non-factual outputs in the model reasoning process, thereby achieving the purpose of enhancing the reliability of large model applications and improving user trust.

[0081] As an optional solution, the method further includes: obtaining different types of seed question data; determining multiple initial question data from the seed question data in batches, and combining the multiple initial question data to obtain multiple sample question data; inputting the multiple sample question data into the target large model to obtain multiple groups of sample answer data, wherein one sample question data among the multiple sample question data corresponds to one group of sample answer data among the multiple groups of sample answer data; generating multiple sample question and answer pairs based on the multiple sample question data and the multiple groups of sample answer data, wherein one sample question and answer pair among the multiple sample question and answer pairs includes one sample question data, one sample answer data and a sample label, and the sample label reflects the association relationship between the one sample answer data and the one sample question data; using the multiple sample question and answer pairs to train the initial large model until the initial large model meets the preset convergence conditions, and then determining the initial large model as the target large model.

[0082] Optionally, in an embodiment of the present application, different types of seed question data refer to a series of high-quality questions prepared in advance, which cover multiple fields, such as finance, technology, geography, and history, etc., and are used to guide the large model to generate diverse factual question and answer data, including but not limited to the textual representation of the questions and the knowledge fields involved. The multiple initial question data determined in batches from the seed question data, and the multiple sample question data obtained by combining these initial question data, are generated through random sampling and combination strategies, aiming to expand the scope and diversity of the question data, and provide material for the subsequent generation of more factual question and answer pairs. Multiple groups of sample answer data are multiple answers output by the large model for the above-mentioned sample question data, forming the basis of the question and answer pair data set.

[0083] It should be noted that the selection and combination of seed question data can be adjusted according to the application scenario and the needs of the data set. For example, the seed question can be a representative question from a real user query, or a classic question in a professional field. The combination strategy can adopt a cross-combination of questions from different fields, or a hierarchical combination based on the complexity of the question. This application does not limit this, as long as it can generate sample question-answer pairs covering a wide range of fields and with rich and diverse content.

[0084] In addition, the setting of convergence conditions can be changed according to the training objectives and performance indicators of the model. For example, the measurement criteria can be set based on the improvement of classification accuracy, the reduction of loss function, or the stability of performance on the test set.

[0085] In an exemplary embodiment, different types of seed question data are first collected, including historical data queries on financial markets, questions about ancient Chinese historical events, questions and answers about geographical knowledge, etc. Next, multiple initial question data are randomly selected from these seed question data in batches, and they are combined into new sample question data using a random combination strategy.

[0086] For example, the initial question data in the fields of finance and history are combined to obtain a question about financial policies in a specific historical period. These sample question data are input into the large model to obtain multiple sets of sample answer data for each question. Then, based on the question data and the corresponding answer data, as well as the label information obtained through the consistency review strategy, multiple sample question-answer pairs are generated. Finally, these sample question-answer pairs are used to train the initial large model until the model meets the preset convergence conditions in key indicators such as classification accuracy and loss function. At this time, the trained model is determined as the target large model for subsequent factual question-answering and hallucination detection tasks.

[0087] Through the embodiments of the present application, different types of seed question data generation, sample question data combination, and sample question and answer pair training strategies are adopted to improve the generalization ability and accuracy of the target large model in the field of factual question and answer. This not only enhances the model's ability in multi-domain knowledge question and answering, but also provides a solid data foundation for the effectiveness of the hallucination detection algorithm, thereby achieving the purpose of improving the model's application value and user experience.

[0088] As an optional scheme, the above-mentioned method of generating multiple sample question and answer pairs based on the above-mentioned multiple sample question data and the above-mentioned multiple groups of sample answer data includes: determining each sample question data in the above-mentioned multiple sample question data as the current sample question data in turn, and performing the following steps on the above-mentioned current sample question data to determine the above-mentioned multiple sample question and answer pairs: determining a group of current sample answer data corresponding to the current sample question data from the above-mentioned multiple groups of sample answer data; determining a basic sample answer data and multiple reference sample answer data from the above-mentioned group of current sample answer data; inputting the above-mentioned current sample question data and the above-mentioned group of current sample answer data into the above-mentioned target large model multiple times to obtain multiple reference sample labels corresponding to each reference sample answer data in the above-mentioned multiple reference sample answer data; based on the above-mentioned multiple reference sample labels, the above-mentioned current sample question data and the above-mentioned basic sample answer data generate a sample question and answer pair.

[0089] Optionally, in an embodiment of the present application, multiple sample question data refer to a series of questions combined in batches by different types of seed question data, each question is designed to test the performance of the large model in factual question and answer, including but not limited to the textual expression of the question, the domain scope of the question, and the complexity of the question. Multiple groups of sample answer data refer to multiple different answers generated by the large model for each sample question data, which reflect the diversity and uncertainty of the model on a given question. Basic sample answer data and multiple reference sample answer data are parts of multiple groups of sample answer data, the basic sample answer data is regarded as a reference benchmark, and the multiple reference sample answer data are used for comparison and analysis with the basic sample answer data.

[0090] It should be noted that the process of inputting each sample problem data into the target large model multiple times to obtain multiple reference sample labels can be achieved using different strategies.

[0091] For example, different decoding strategies, different model temperature parameters, or different prompting methods can be used to increase the diversity of the responses generated by the model each time.

[0092] At the same time, the generation and analysis of reference sample labels can rely on a variety of model evaluation techniques, including but not limited to model consistency checks, external knowledge base verification, or human expert reviews. This application does not limit this, as long as valid sample question-answer pairs can be generated based on these labels and answers.

[0093] In an exemplary embodiment, a question about ancient Chinese history is selected from multiple sample question data: "In which year did Qin Shihuang unify the six kingdoms?" as the current sample question data. Then, from multiple groups of sample answer data generated by the large model, a group of current sample answer data corresponding to the question is determined, including a basic sample answer data: "Qin Shihuang unified the six kingdoms in 221 BC" and several reference sample answer data, such as: "Qin Shihuang unified the six kingdoms in 220 BC", "Qin Shihuang unified the six kingdoms in 222 BC", etc. Then, the current sample question data and a group of current sample answer data are input into the target large model multiple times to obtain multiple reference sample labels corresponding to each reference sample answer data, which are consistent, neutral or inconsistent. Finally, based on these reference sample labels and comparison with the basic sample answer data, a sample question-answer pair is generated, which includes questions, basic answers, and sample labels reflecting the accuracy or consistency of the answers.

[0094] Through the embodiments of the present application, a strategy of multiple input of sample question data and sample answer data into the target large model to generate reference sample labels, and then constructing sample question and answer pairs, achieves the technical effect of enhancing the accuracy detection of factual questions and answers of the large model and the recognition of model hallucinations, achieves the purpose of improving the model training efficiency and the accuracy of the hallucination detection algorithm, and ensures that the model can provide more reliable and accurate answers in practical applications.

[0095] As an optional scheme, the above-mentioned generation of a sample question and answer pair based on the above-mentioned multiple reference sample labels, the above-mentioned current sample question data and the above-mentioned basic sample answer data includes: respectively counting the number of occurrences of each sub-label in the above-mentioned multiple reference sample labels corresponding to the above-mentioned each reference sample answer data, and determining the above-mentioned sub-label with the largest number of occurrences as the target reference label of the above-mentioned reference sample answer data corresponding thereto; determining the above-mentioned sub-label with the largest number of occurrences in multiple target reference labels as the target basic label, wherein the above-mentioned target basic label corresponds to the above-mentioned basic sample answer data; and generating a sample question and answer pair based on the above-mentioned current sample question data, the above-mentioned basic sample answer data and the above-mentioned target basic label.

[0096] Optionally, in an embodiment of the present application, multiple reference sample labels refer to a set of labels for each reference sample answer data generated by a model or manual review after multiple inputs of sample question data and sample answer data into a target large model. Each sub-label is a basic unit that constitutes a reference sample label set, and they represent the evaluation results of the model answer, including but not limited to "consistent", "neutral" and "inconsistent". The target reference label and the target basic label are determined based on statistical analysis. The target reference label is used to reflect the main evaluation results of each reference sample answer data, while the target basic label is based on the voting results of multiple target reference labels, reflecting the comprehensive evaluation of the basic sample answer data.

[0097] It should be noted that when the label with the most occurrences does not exceed a certain proportion, additional rules can be set to process such data. For example, a minimum proportion threshold can be set. When no label reaches this proportion, the sample data is marked as "uncertain" or "requires manual review". This application does not limit this, as long as the sample question and answer pairs can be generated reasonably.

[0098] In an exemplary embodiment, assume that there are 5 reference sample answer data for the same question: "Which is the largest lake in China?" and a basic sample answer data: "The largest lake in China is Qinghai Lake". After inputting these data into the target large model multiple times, multiple reference sample labels are obtained for each reference sample answer data.

[0099] For example, for the first reference sample answer data: "The largest lake in China is Taihu Lake", the obtained label set is {"inconsistent", "neutral", "inconsistent", "consistent", "inconsistent"}. By counting the number of occurrences of each sub-label in these label sets, the sub-label with the largest number of occurrences is determined to be "inconsistent", which becomes the target reference label corresponding to the first reference sample answer data.

[0100] Further, the above process is repeated to determine a target reference label for each reference sample answer data. Finally, based on the number of occurrences of all target reference labels, the sub-label with the largest number of occurrences is determined as the target basic label of the sample question and answer pair. If "inconsistent" appears the most times among all target reference labels, "inconsistent" is determined as the target basic label, and together with the current sample question data and the basic sample answer data, a sample question and answer pair marked as "inconsistent" is generated.

[0101] Through the embodiments of the present application, a strategy of sub-label statistics and voting to determine the target reference label and the target basic label is adopted, which achieves the technical effect of quantitatively evaluating the hallucination phenomenon answered by a large model, and achieves the purpose of improving the interpretability and credibility of the model output results, ensuring that in the subsequent model training and optimization process, more targeted feedback information can be obtained, thereby improving the overall performance of the model and user satisfaction.

[0102] For example, the current hallucination detection technology is mainly divided into two categories: consistency-based methods and detection-based methods. The consistency-based method detects non-factual generation by comparing the consistency of the content generated by the model multiple times. The core assumption of this method is that if a large language model cannot give consistent answers to the same question multiple times, then there is a high probability that hallucinations have occurred. The detection-based method believes that the hidden representation of the model contains certain attributes of the generated content, which can be extracted and classified through a lightweight model. Existing detection-based methods often require large-scale annotated data, but there is a lack of corresponding hallucination detection datasets in the Chinese field.

[0103] Furthermore, existing consistency-based methods require multiple sampling of model answers, and then judge whether the model output has hallucinations based on the model's multiple answers. This results in the time required for hallucination detection being several times the time required for the model to answer questions normally, and it cannot be applied to actual business scenarios at all. In addition, since the research on hallucination detection technology is still in its early stages, most of the current research is focused on English scenarios, and most experiments are based on English open source datasets. There are still deficiencies in the research on hallucination detection in Chinese scenarios. Existing hallucination detection methods based on the internal state of the model often only perform detection based on one state, and the input features of the classifier are relatively simple and have poor robustness.

[0104] In summary, in order to solve the problems existing in the existing methods, the embodiments of the present application propose a hallucination detection algorithm based on Chinese, with real-time performance, and using multiple classifiers to detect multiple states within the model.

[0105] In an exemplary embodiment, the present application embodiment may include the following steps:

[0106] S1, first use the large model to generate a large number of factual question-answer pairs, and perform multiple sampling on each question to obtain multiple answers;

[0107] S2, then use the big model to conduct multiple rounds of voting on the consistency of the answer content to obtain the labels of the question and answer pairs, and finally obtain the labeled factual question and answer data;

[0108] S3 uses an integrated learning method to detect hallucinations. Two classifiers are used in the detection process. Specifically:

[0109] One of the classifiers (the first classifier) ​​uses the hidden state of the model's intermediate layer output as a feature (the first output feature), and the other classifier (the second classifier) ​​uses the output of the first 28 special attention heads that are most relevant to the model's authenticity as a feature (the second output feature). Both classifiers are trained on the factual question-answer data obtained by S2 (the sample question-answer pairs).

[0110] S4, apply the trained classifier to the hallucination detection task.

[0111] Among them, the generation of factual question and answer data includes three parts: generating questions, generating answers and consistency review. Building and training classifiers is divided into three parts: building classifiers, extracting features and training classifiers.

[0112] In an exemplary embodiment, Figure 3 is a schematic diagram of an optional model hallucination detection method according to an embodiment of the present application. The overall process is as follows: Figure 3 shown.

[0113] Exemplarily, the factual question-answering data is a Chinese dataset, and the generation of the Chinese dataset may include but is not limited to the following steps:

[0114] S1, generate questions. In the stage of generating questions, use prompt engineering to guide the model to output diverse factual questions and answers. Completed based on the Qwen2-7B-Instruct model. Before generating questions, it is necessary to manually select high-quality seed questions that include multiple fields. The seed questions selected by the present invention include fields such as finance, technology, geography, and history, ensuring the diversity of seed questions. Then a prompt strategy of random sampling and random combination is adopted, that is, 5 questions are randomly selected from the seed questions each time and randomly combined as prompts to guide the model to generate similar questions. Through this random strategy, the model can output factual questions in different fields very well.

[0115] S2, generate answers, use the previously generated questions as input to the large language model, and prompt the large language model to generate multiple responses for subsequent consistency checks. In order to make the model's answers more diverse, the following two strategies are used in this stage: First, temperature is an important hyperparameter in the large model, and its size will affect the diversity and randomness of the model output. Therefore, the embodiment of the present application selects the vanilla sampling method with a relatively high temperature as the decoding strategy of the model at this stage.

[0116] First, the embodiment of the present application designs multiple prompt templates at this stage, and uses different prompt templates to guide the large language model to answer questions in different ways. Through the organic combination of these two strategies, the large model can output multiple significantly different answers to the same question. After determining the generation strategy, the large model is used in this stage to output multiple answers to each question, and the question-answer pair dataset {(q,{r i})}, where q represents the question, {r i} represents multiple answers corresponding to each question, K = |{r i}| is a hyperparameter of the experiment. In the embodiment of the present application, K=5 can be selected for data generation.

[0117] S3: In this stage, the large model is used to analyze the question-answer pairs {(q,{r i})} to conduct consistency review on multiple answers to the same question, generate labels for the question-answer pairs, and then obtain a labeled dataset. This stage is mainly divided into the following three steps:

[0118] S3-1, review stage: First, the K answers to each question in the data set are divided into a basic answer and multiple reference answers, that is, (q, {r i}) is divided into (q,r0,{r j}), where r0 represents the basic answer, r j Indicates a reference answer.

[0119] S3-2, when reviewing, select a reference answer each time and input it into the big model together with the question and the basic answer, so that the big model can judge the consistency of the content of the basic answer and the reference answer.

[0120] Specifically, prompts are used to allow the large model to output only three words, Consistent, Neutral, and Inconsistent, each time during the review process, so as to obtain a pseudo-label for each reference answer.

[0121] The above process is repeated, and after N rounds of review, each reference answer has N pseudo labels.

[0122] S4, voting stage: count the number of various pseudo labels (i.e., consistent, neutral, and inconsistent) in each reference answer, and select the pseudo label with the largest number as the label of each reference answer. In the voting stage, controversial answers will be filtered out, that is, if no single label (consistent, neutral, inconsistent) receives more than 50% of the votes, the current data will be filtered out. The data set obtained after the voting stage can be expressed as {(q,{r0,(r1,l1)...(r j ,l j )})}, where l j Represents its corresponding reference answer r j .

[0123] S5, vote again: vote based on the label of each reference answer, and select the label with the largest number of all reference answers corresponding to each basic answer as the label of the basic answer. If no single label receives more than 50% of the votes or the label with the most votes is neutral, the current data is filtered out. Finally, the factual label of each basic answer is obtained, and the final data set can be expressed as {(q,r,l)}.

[0124] For example, in the embodiment of the present application, in the stage of constructing and training a classifier, the following steps may be included but are not limited to:

[0125] S1, build a classifier model, use a multi-layer perceptron as the classifier model, the classifier receives the internal state generated by the large model reasoning process as input, and the output is whether the current answer is hallucination.

[0126] The two classifiers can use the same or different model structures. The input dimension of the classifier model is the model dimension of the Qwen2-7B-Instruct large model, the output dimension is 2, and relu is used as the activation function. The formula of the relu activation function is f(x)=max(0,x).

[0127] S2, feature extraction, Figure 4is a schematic diagram of another optional method for detecting model hallucination according to an embodiment of the present application, Figure 5 is a schematic diagram of another optional method for detecting model hallucination according to an embodiment of the present application, such as Figure 4 , Figure 5 As shown in the figure, this stage mainly converts the dataset generated in the previous section into features that can be used for model training. In order to improve the detection effect and robustness of large model hallucination detection, two classifiers are used to detect the two states inside the model at the same time. Therefore, two types of feature data must be extracted at the same time in this stage. They are the hidden state F1 output by the middle layer during model reasoning and the output F2 of the authenticity attention head during the model reasoning process.

[0128] Specifically, to obtain the hidden state F1 of the middle layer output, you only need to set the output_hidden_states parameter (the above-mentioned output hidden state parameter) to True during large model inference.

[0129] The process of obtaining the output feature F2 of the authenticity attention head can be divided into the following two steps:

[0130] S1, get the position of the authenticity attention head. The Qwen2-7B-Instruct model has 28 layers, each layer has 28 attention heads, and a total of 784 attention heads. The goal of the current step is to find the attention head that is most relevant to the authenticity of the model's answer from all the attention heads.

[0131] S2, in the process of finding the authenticity attention head, we first use the output of each attention head as a feature, and train 784 classifiers on 784 attention heads based on the previously generated dataset and constructed classifier. Then, we rank the output features of different attention heads based on their performance on the classification model, and get the top 28 attention heads that are most relevant to authenticity.

[0132] Then, the outputs of the first 28 authenticity attention heads are concatenated to obtain feature F_2. Since the feature extraction process requires the use of a large model and the entire process is time-consuming, at this stage, the extracted features can be converted into numpy arrays and stored in files for subsequent use.

[0133] S3, training the classifier model. In this stage, the cross entropy loss function is used to calculate the model loss. The cross entropy loss function formula is as follows:

[0134]

[0135] Among them, is y i Label value, y i ′is the predicted value, and n is the number of samples in the training set. During training, the Adam optimizer is used to optimize the model parameters, and the trained classifier model is obtained after training for 10 time steps using a learning rate of 2e-4.

[0136] S5, after obtaining the output results of the two classifier models, it can be determined whether hallucination occurs during the model answering process according to the output results. Specifically, when both classifiers determine that hallucination occurs, the present invention will determine that hallucination does exist. If one classifier believes that hallucination occurs and the other believes that hallucination does not occur, it is determined that there is a risk of hallucination. If both classifiers believe that hallucination does not occur, it is determined that there is no hallucination. Figure 6 is a schematic diagram of another optional model hallucination detection method according to an embodiment of the present application, and the specific process is as follows Figure 6 shown.

[0137] In general, the embodiment of the present application successfully generated a Chinese factual question-and-answer data set with the help of the powerful generation capability and consistency review strategy of the large model. On this basis, a hallucination classifier was trained for the Chinese factual question-and-answer scenario, and the hallucination classifier finally trained can effectively detect the non-factual content output by the large model; using the concept of ensemble learning, two classifier models were trained in combination with different states inside the large model, and the output results of the two classifier models were combined for decision making, which not only significantly improved the robustness of the detection method but also effectively improved the accuracy of hallucination detection; using a multi-layer perceptron as the classifier model, the number of parameters of the entire classifier model is only one ten-thousandth of the number of parameters of the large model, and the hallucination detection time is much less than the normal inference time of the model, thereby enabling real-time hallucination detection.

[0138] Through the embodiment of the present application, the hallucination detection accuracy in the Chinese scene is higher, which can be better combined with the use process of the target large model, achieving real-time hallucination detection, and can be applied to actual user usage scenarios to achieve hallucination detection without user perception. Comprehensive decision-making is made based on the results of two different classifiers, so that the detection accuracy is better and the detection results are more robust.

[0139] It is understandable that in the specific implementation of this application, related data such as user information is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0140] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0141] According to another aspect of the embodiments of the present application, a model hallucination detection device for implementing the above-mentioned model hallucination detection method is also provided. Figure 7 As shown, the device comprises:

[0142] A determination module 702 is used to input the problem data into the target large model, and determine a first output feature and a second output feature, wherein the first output feature represents a hidden state output feature of an intermediate layer of the target large model, and the second output feature represents an output feature obtained by concatenating output features of M attention heads of the target large model, where the M attention heads are attention heads that meet the target output condition, and M is a positive integer;

[0143] A processing module 704 is used to input the first output feature into a first classifier to obtain a first detection sub-result, and input the second output feature into a second classifier to obtain a second detection sub-result, wherein both the first detection sub-result and the second classifier are used to indicate whether the target large model has model hallucination;

[0144] The generating module 706 is used to generate a hallucination prompt message when both the first detection sub-result and the second detection sub-result indicate that the target large model has model hallucination, wherein the hallucination prompt message indicates that the model parameters of the target large model need to be adjusted.

[0145] As an optional solution, the above-mentioned device is also used to: generate a hallucination risk warning message when any one of the first detection sub-result and the second detection sub-result indicates that the target large model has model hallucination, wherein the hallucination risk warning message indicates that it is necessary to detect the model parameters to determine whether the model parameters need to be adjusted; and generate answer data output by the target large model for the question data when both the first detection sub-result and the second detection sub-result indicate that the target large model does not have model hallucination.

[0146] As an optional solution, the above-mentioned device is also used for at least one of the following: setting the output hidden state parameter of the target large model to true to obtain the first output feature; obtaining the output feature of each attention head in the M attention heads, and splicing the output features to obtain the second output feature.

[0147] As an optional solution, the above-mentioned device is also used to: obtain the output features of each attention head in the M attention heads, and before splicing the output features to obtain the second output features, input the training set data into the target large model to obtain the output features of each attention head in the multiple attention heads; use the output features of each attention head to train a group of classifiers respectively, wherein a classifier in a group of classifiers corresponds to one or more attention heads in the multiple attention heads; input the verification set data into the target large model to determine the classification accuracy parameters of each classifier in the group of classifiers; sort the initial classification accuracy parameters of a group of classifiers according to the size of the values ​​to obtain a group of classification accuracy parameters; determine the classification accuracy parameters in the group of classification accuracy parameters whose values ​​are greater than or equal to the accuracy threshold as the target classification accuracy parameters; and determine the attention heads corresponding to the target classification accuracy parameters as M attention heads.

[0148] As an optional solution, the above-mentioned device is also used to: obtain different types of seed question data; determine multiple initial question data from the seed question data in batches, and combine the multiple initial question data to obtain multiple sample question data; input the multiple sample question data into the target large model to obtain multiple groups of sample answer data, wherein one sample question data in the multiple sample question data corresponds to one group of sample answer data in the multiple groups of sample answer data; generate multiple sample question and answer pairs based on the multiple sample question data and the multiple groups of sample answer data, wherein one sample question and answer pair in the multiple sample question and answer pairs includes a sample question data, a sample answer data and a sample label, and the sample label reflects the association relationship between a sample answer data and a sample question data; use the multiple sample question and answer pairs to train the initial large model until the initial large model meets the preset convergence conditions, and then determine the initial large model as the target large model.

[0149] As an optional solution, the above-mentioned device is used to generate multiple sample question and answer pairs based on multiple sample question data and multiple groups of sample answer data in the following manner: each sample question data in the multiple sample question data is determined as the current sample question data in turn, and the following steps are performed on the current sample question data to determine multiple sample question and answer pairs: a group of current sample answer data corresponding to the current sample question data is determined from the multiple groups of sample answer data; a basic sample answer data and multiple reference sample answer data are determined from a group of current sample answer data; the current sample question data and a group of current sample answer data are input into the target large model multiple times to obtain multiple reference sample labels corresponding to each reference sample answer data in the multiple reference sample answer data; based on the multiple reference sample labels, the current sample question data and the basic sample answer data generate a sample question and answer pair.

[0150] As an optional solution, the above-mentioned device is used to generate a sample question and answer pair based on multiple reference sample labels, current sample question data and basic sample answer data in the following manner: respectively counting the number of occurrences of each sub-label in the multiple reference sample labels corresponding to each reference sample answer data, and determining the sub-label with the largest number of occurrences as the target reference label of the reference sample answer data corresponding thereto; determining the sub-label with the largest number of occurrences in the multiple target reference labels as the target basic label, wherein the target basic label corresponds to the basic sample answer data; and generating a sample question and answer pair based on the current sample question data, the basic sample answer data and the target basic label.

[0151] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0152] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0153] According to one aspect of the present application, a computer program product is provided. The computer program product includes a computer program.

[0154] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0155] Figure 8 The structure block diagram of a computer system for implementing an electronic device according to an embodiment of the present application is schematically shown.

[0156] It should be noted that Figure 8 The computer system 800 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0157] like Figure 8As shown, the computer system 800 includes a central processing unit 801 (CPU), which can perform various appropriate actions and processes according to the program stored in the read-only memory 802 (ROM) or the program loaded from the storage part 808 to the random access memory 803 (RAM). Various programs and data required for system operation are also stored in the random access memory 803. The central processing unit 801, the read-only memory 802 and the random access memory 803 are connected to each other through a bus 804. The input / output interface 805 (Input / Output interface, i.e., I / O interface) is also connected to the bus 804.

[0158] The following components are connected to the input / output interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a local area network card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed so that a computer program read therefrom is installed into the storage section 808 as needed.

[0159] In particular, according to an embodiment of the present application, the process described in each method flow chart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program contains a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the central processor 801, various functions defined in the system of the present application are executed.

[0160] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the central processor 801, various functions provided by the embodiment of the present application are performed.

[0161] According to another aspect of the embodiment of the present application, an electronic device for implementing the above-mentioned model hallucination detection method is also provided. The electronic device may be Figure 1 The terminal device or server shown in the figure. This embodiment is described by taking the electronic device as a terminal device as an example. Fig. 9 As shown, the electronic device includes a memory 902 and a processor 904. The memory 902 stores a computer program, and the processor 904 is configured to execute the steps in any of the above method embodiments through the computer program.

[0162] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0163] Optionally, in this embodiment, the above-mentioned processor can be configured to execute the methods in each embodiment of the present application through a computer program.

[0164] Alternatively, a person skilled in the art may understand that: Fig. 9 The structure shown is for illustration only. Fig. 9 The structure of the electronic device is not limited. Fig. 9 More or fewer components (such as network interfaces, etc.) as shown in, or with Fig. 9 Different configurations are shown.

[0165] Among them, the memory 902 can be used to store software programs and modules, such as the program instructions / modules corresponding to the detection method and device of model hallucination in the embodiment of the present application. The processor 904 executes various functional applications and data processing by running the software programs and modules stored in the memory 902, that is, to implement the above-mentioned detection method of model hallucination. The memory 902 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 902 may further include a memory remotely arranged relative to the processor 904, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Among them, the memory 902 can be specifically, but not limited to, used to store the above-mentioned sample question-answer peer information. As an example, such as Fig. 9 As shown, the memory 902 may include, but is not limited to, the determination module 702, the processing module 704, and the generation module 706 in the model hallucination detection device. In addition, it may also include, but is not limited to, other module units in the model hallucination detection device, which will not be repeated in this example.

[0166] Optionally, the transmission device 906 is used to receive or send data via a network. Specific examples of the above-mentioned network may include a wired network and a wireless network. In one example, the transmission device 906 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 906 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0167] In addition, the electronic device further comprises: a display 908 for displaying the detection result of the model illusion; and a connection bus 910 for connecting various module components in the electronic device.

[0168] In other embodiments, the terminal device or server may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting the multiple nodes through network communication. The nodes may form a peer-to-peer network, and any form of computing device, such as a server, terminal or other electronic device, may become a node in the blockchain system by joining the peer-to-peer network.

[0169] According to one aspect of the present application, a computer-readable storage medium is provided, and a processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the model hallucination detection method provided in various optional implementations of the above-mentioned model hallucination detection aspects.

[0170] Optionally, in this embodiment, the above-mentioned computer-readable storage medium can be configured to store data for executing the methods in various embodiments of the present application.

[0171] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0172] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0173] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for one or more electronic devices to execute all or part of the steps of the methods described in each embodiment of the present application.

[0174] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0175] In the several embodiments provided in the present application, it should be understood that the disclosed application can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0176] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0177] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0178] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for detecting model hallucination, characterized in that: include: Input the problem data into the target large model, and determine the first output feature and the second output feature, wherein the first output feature represents the hidden state output feature of the middle layer of the target large model, and the second output feature represents the output feature obtained by concatenating the output features of M attention heads of the target large model, wherein the M attention heads are attention heads that meet the target output condition, and M is a positive integer; Inputting the first output feature into a first classifier to obtain a first detection sub-result, and inputting the second output feature into a second classifier to obtain a second detection sub-result, wherein the first detection sub-result and the second classifier are both used to indicate whether the target macro model has model hallucination; In a case where both the first detection sub-result and the second detection sub-result indicate that model hallucination occurs in the target large model, a hallucination prompt message is generated, wherein the hallucination prompt message indicates that model parameters of the target large model need to be adjusted.

2. The method according to claim 1, characterized in that The method further comprises: In the case where any one of the first detection sub-result and the second detection sub-result indicates that the target large model has the model hallucination, generating a hallucination risk prompt message, wherein the hallucination risk prompt message indicates that the model parameters need to be detected to determine whether the model parameters need to be adjusted; When both the first detection sub-result and the second detection sub-result indicate that the target large model does not exhibit the model hallucination, answer data output by the target large model in response to the question data is generated.

3. The method according to claim 1, characterized in that The method further comprises at least one of the following: Setting the output hidden state parameter of the target large model to true to obtain the first output feature; Obtain output features of each of the M attention heads, and concatenate the output features to obtain the second output feature.

4. The method according to claim 3, characterized in that Before obtaining the output feature of each of the M attention heads and concatenating the output features to obtain the second output feature, the method further includes: Input the training set data into the target large model to obtain output features of each of the multiple attention heads; Using the output features of each of the attention heads, respectively, a group of classifiers are trained, wherein a classifier in the group of classifiers corresponds to one of the multiple attention heads; Inputting the validation set data into the target large model to determine the classification accuracy parameter of each classifier in the set of classifiers; Sorting the initial classification accuracy parameters of the group of classifiers according to their values ​​to obtain a group of classification accuracy parameters; Determine a classification accuracy parameter whose value is greater than or equal to the accuracy threshold in the set of classification accuracy parameters as a target classification accuracy parameter; The attention heads corresponding to the target classification accuracy parameter are determined as the M attention heads.

5. The method according to claim 1, characterized in that: The method further comprises: Get different types of seed question data; Determining a plurality of initial question data from the seed question data in batches, and combining the plurality of initial question data to obtain a plurality of sample question data; Inputting the plurality of sample question data into the target macro model to obtain a plurality of groups of sample answer data, wherein one sample question data in the plurality of sample question data corresponds to a group of sample answer data in the plurality of groups of sample answer data; Generate a plurality of sample question-answer pairs according to the plurality of sample question data and the plurality of groups of sample answer data, wherein one sample question-answer pair among the plurality of sample question-answer pairs comprises one sample question data, one sample answer data and a sample label, and the sample label reflects the association relationship between the one sample answer data and the one sample question data; The initial large model is trained using the multiple sample question-answer pairs until the initial large model meets a preset convergence condition, and then the initial large model is determined as the target large model.

6. The method according to claim 5, characterized in that The generating a plurality of sample question-answer pairs according to the plurality of sample question data and the plurality of groups of sample answer data comprises: Each sample question data in the plurality of sample question data is sequentially determined as current sample question data, and the following steps are performed on the current sample question data to determine the plurality of sample question-answer pairs: Determine a set of current sample answer data corresponding to the current sample question data from the multiple sets of sample answer data; Determining a basic sample answer data and a plurality of reference sample answer data from the set of current sample answer data; Inputting the current sample question data and the set of current sample answer data into the target large model multiple times to obtain multiple reference sample labels corresponding to each reference sample answer data in the multiple reference sample answer data; Based on the multiple reference sample labels, the current sample question data and the basic sample answer data generate a sample question-answer pair.

7. The method according to claim 6, characterized in that The step of generating a sample question-answer pair based on the multiple reference sample labels, the current sample question data and the basic sample answer data comprises: Counting the number of occurrences of each sub-label in the multiple reference sample labels corresponding to each reference sample answer data, and determining the sub-label with the largest number of occurrences as the target reference label of the reference sample answer data corresponding thereto; Determine the sub-tag with the largest number of occurrences among the multiple target reference tags as a target basic tag, wherein the target basic tag corresponds to the basic sample answer data; A sample question-answer pair is generated according to the current sample question data, the basic sample answer data and the target basic label.

8. A device for detecting model hallucination, characterized in that: include: A determination module, used to input the problem data into the target large model, and determine a first output feature and a second output feature, wherein the first output feature represents a hidden state output feature of an intermediate layer of the target large model, and the second output feature represents an output feature obtained by concatenating output features of M attention heads of the target large model, wherein the M attention heads are attention heads that meet the target output condition, and M is a positive integer; a processing module, configured to input the first output feature into a first classifier to obtain a first detection sub-result, and input the second output feature into a second classifier to obtain a second detection sub-result, wherein the first detection sub-result and the second classifier are both used to indicate whether the target macro model has model hallucination; A generation module is used to generate a hallucination prompt message when both the first detection sub-result and the second detection sub-result indicate that the target large model has model hallucination, wherein the hallucination prompt message indicates that the model parameters of the target large model need to be adjusted.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein the computer program can be executed by an electronic device to perform the method described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

11. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 7 through the computer program.

Images