A large model inference process detection method, device, medium and equipment

Abstract

The embodiment of the specification discloses a large model output result detection method. The method is based on training samples labeled with whether the output result is correct, and determines the activation state of neurons in the inference process of the large model LLM to be detected, to obtain the relationship between the correctness of the output result and the activation state of the neurons, and train a detection classifier. Then, when the LLM to be detected executes task input data, the activation state of the neurons is obtained, and the correctness of the LLM output result is detected by the detection classifier, that is, whether hallucination occurs. The method links the relationship between the activation state of the LLM neurons and the occurrence of hallucination of the LLM, so as to identify whether the LLM hallucinates, without paying attention to the inference process and the output result of the LLM, and identifying whether the LLM hallucinates from the bottom of the model.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a method, apparatus, storage medium and device for detecting large model reasoning processes. Background Technology

[0002] With the development of artificial intelligence (AI) technology, large language models (LLM) have been widely applied in various fields.

[0003] However, LLM may suffer from problems such as lack of knowledge, knowledge errors, or outdated knowledge due to limitations in training data. These problems may cause LLM to experience "illusions" when performing tasks, thereby reducing the accuracy and reliability of LLM in performing tasks.

[0004] Therefore, detecting "illusions" in the reasoning process of LLM and preventing LLM from outputting incorrect answers has become an urgent problem to be solved. Thus, this specification provides a method for detecting large model reasoning processes. Summary of the Invention

[0005] This specification provides a method, apparatus, storage medium, and electronic device for detecting large model output results, in order to partially solve the problems existing in the prior art.

[0006] The embodiments in this specification adopt the following technical solutions:

[0007] This specification provides a method for detecting the output results of a large model. The method includes:

[0008] Obtain training samples for training a large model, the training samples containing annotations indicating whether the model output is correct;

[0009] Based on the input training samples, the activation state of neurons in the large model to be detected is determined as the activation state corresponding to the training samples;

[0010] Using the activation state corresponding to the training sample as input, and labeling whether the output result is correct, a detection classifier is trained.

[0011] In response to the activation state of neurons executing the business process of the large model to be detected, the correctness of the output result of the large model to be detected is detected by the detection classifier.

[0012] This specification provides a large model output result detection device, the device comprising:

[0013] The acquisition module is used to acquire training samples for training a large model, wherein the training samples contain annotations indicating whether the model output results are correct.

[0014] The inference module is used to determine the activation state of neurons in the large model to be detected based on the input training sample, and use it as the activation state corresponding to the training sample.

[0015] The training module is used to train a detection classifier by taking the activation state corresponding to the training sample as input and labeling whether the output result is correct.

[0016] The detection module is used to detect the correctness of the output results of the large model under test by means of the detection classifier in response to the activation state of the neurons executing the business process of the large model under test.

[0017] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for detecting the output results of a large model.

[0018] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the above-described method for detecting the output results of a large model.

[0019] The above-described at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects:

[0020] This specification discloses a method for detecting the output results of a large model. This method uses training samples labeled with information on the correctness of the output results, and determines the activation states of neurons in the Large LLM (Limited Learning Model) during inference. It obtains the relationship between the correctness of the output results and the neuron activation states, and uses this relationship to train a detection classifier. When the LLM executes task input data, the activation states of neurons are acquired, and the detection classifier detects the correctness of the LLM's output results, i.e., whether "hallucinations" have occurred. This method identifies whether the LLM is experiencing hallucinations by linking the activation states of LLM neurons with the occurrence of hallucinations. It does not require attention to the LLM's inference process and output results; it identifies whether the LLM is experiencing hallucinations from the model's underlying layers, thus avoiding the execution of business operations based on output results that indicate "hallucinations." Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings:

[0022] Figure 1A flowchart for detecting the output results of a large model provided in the embodiments of this specification;

[0023] Figure 2 This is a schematic diagram illustrating the recording of neuron activation states provided in the embodiments of this specification;

[0024] Figure 3 This is a schematic diagram of the large model output result detection process provided in the embodiments of this specification;

[0025] Figure 4 A schematic diagram of a large model output result detection device provided in the embodiments of this specification;

[0026] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this specification. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0028] Large Language Model (LLM), also known simply as a large model, is a model that performs logical reasoning on input natural language text and outputs a natural language description that users can understand. Although LLM can generate fluent and logically sound content, its output is not always based on facts, which is known as model "hallucination".

[0029] Existing methods to address the "illusion" phenomenon in LLMs include incorporating knowledge into the LLM through Supervised Fine-Tunneling (SFT) to address the occurrence of factual "illusion" situations. Alternatively, based on labeled training samples that induce "illusions," the LLM learns under what circumstances "illusions" occur, and then corrects the output for "illusions" in real-world business scenarios. However, the differences between various LLMs clearly mean that the training samples may not perform consistently across different LLMs, necessitating adjustments to the training samples.

[0030] However, this also leads to the need for a large number of targeted training samples, resulting in high training costs and requiring significant time for fine-tuning the LLM. Furthermore, fine-tuning the LLM can cause conflicts between training tasks, making it difficult to balance training effectiveness. For example, there might be a conflict between training tasks that aim to produce accurate LLM outputs and training tasks that aim to prevent the LLM from producing "illusion" results.

[0031] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0032] Figure 1 A flowchart for detecting the output results of a large model, provided in the embodiments of this specification, specifically includes the following steps:

[0033] S100: Obtain training samples for training a large model, the training samples containing annotations indicating whether the model output is correct.

[0034] In the embodiments described in this specification, the following are employed: Figure 1 The device used for large model output result detection using the method shown can be any electronic device, such as a computer, server, or server cluster consisting of multiple servers. For ease of description, the following explanation uses a server as an example.

[0035] To detect whether the LLM output exhibits "illusion," the server needs to obtain training samples labeled with and without "illusion." Specifically, the server can first obtain training samples for training the LLM from an open-source database. These training samples include: model input, output, and a label indicating whether "illusion" occurred. Model input refers to the data input to the LLM. Output refers to the responses the LLM should produce in response to the model input. The label indicating whether "illusion" occurred indicates when the LLM, based on the model input, performs logical reasoning and determines the output of the training sample, exhibits model "illusion." Of course, since the number of training samples labeled with "illusion" may be small, the training samples required in this embodiment can also be obtained by modifying samples without "illusion" labels.

[0036] For example, obtain a large number of LLM training samples and add labels indicating that no "hallucination" occurred. Then input the training samples into the LLM to be trained, and use the erroneous output of the LLM to be trained as the output of the training samples, adding labels indicating that "hallucination" occurred.

[0037] Of course, in the embodiments of this specification, there are no restrictions on the annotation format for whether "hallucination" occurs; it can be natural language description text or different identifiers. For example, 0 indicates that "hallucination" did not occur, and 1 indicates that "hallucination" occurred. The specific format can be set as needed, and this specification does not impose any limitations. Furthermore, the model input and output results included in the above training samples are all natural language description text.

[0038] S102: Determine the activation state of neurons in the large model to be detected based on the input training sample, and use it as the activation state corresponding to the training sample.

[0039] After obtaining the training samples, the server can input them into the LLM to be tested and record the activation states of each neuron in the LLM during the inference process to obtain the output result. These activation states are then used as the corresponding activation states for the training samples. The LLM can be pre-deployed on the server or on other devices. If deployed on other devices, the server can send the business logic process to those devices, allowing them to input the received business logic process into the LLM. The following explanation uses the deployment of the LLM on the server as an example.

[0040] Of course, the neurons in the LLM described in the embodiments of this specification are mathematical functions in the LLM, that is, the basic computational units that constitute the neural network. Existing neural network models, that is, the basic computational principle of LLM, achieve complex pattern recognition and reasoning capabilities by simulating the "input-processing-output" behavior of biological neurons. Structurally, each basic computational function can be considered a neuron, or a set of computational functions composed of multiple basic mathematical functions can be considered a neuron. This specification does not impose specific restrictions on the structure of neurons; neurons can be determined according to the model structure designed during the construction of the LLM, or the most basic mathematical functions can be used as neurons, set as needed. Generally, the neurons designed during the construction of the LLM are sufficient to implement the embodiments of this specification, and will be used as an example in the following explanations.

[0041] For different model inputs, the activation states of neurons within the LLM model differ when inferring output results. Recording the activation states of neurons in the LLM corresponding to different training samples can characterize how the LLM obtains the output results. Then, by associating different inference representations with whether the output results exhibit "illusions," it can be determined under what inference representations the LLM to be tested exhibits "illusions." For example, when the LLM fails to obtain the correct output result, it will always rely on several methods to generate an output result that "looks like the answer," i.e., an "illusion" occurs. The activation states of neurons exhibiting these "looks like the answer" output results must have commonalities. Therefore, as long as the commonalities of neuron activation states under these "illusion" conditions can be determined, methods for identifying "illusions" can be provided in subsequent steps.

[0042] Furthermore, in the embodiments of this specification, for the neurons of LLM, regardless of the type of data input to the neurons, the final activation state of the corresponding neurons is generally discrete. Typically, the neuron is either activated and continues to output data to the next neuron, or it is not activated and does not output data. Therefore, the activation states of the neurons in the LLM recorded by the server can simply record whether each neuron is activated. By using simpler information to characterize the inference process of LLM, it is easier to determine the commonalities in the activation states of neurons for different training samples.

[0043] Alternatively, the server can also record the specific activation state of neurons in detail. For example, it can record what input information activated them or what input information did not activate them. This allows subsequent steps to obtain a deeper understanding of the connection between neuronal activation states and "hallucinations" based on richer information, increasing the accuracy of detecting "hallucinations" in LLM.

[0044] Figure 2 This is a schematic diagram illustrating the recording of neuron activation states, provided as an embodiment of this specification. Figure 2 The diagram below illustrates the structure of a neural network model, showing the data transmission and neuron states after a training sample is input into the model. Dark dots represent activated neurons, and light dots represent inactive neurons. After the model outputs its results, it records which neurons were activated, such as... Figure 2 The diagram uses a~d to represent a 4-layer model structure. Each neuron is represented by a number from top to bottom, i.e., a1~a4, b1~b6, and so on. The activation states of neurons can be determined as (a1, a2, a3, a4, b1, b2, b5, c4, c5, d1, d2) as the activation states corresponding to the training samples.

[0045] S104: Using the activation state corresponding to the training sample as input, and labeling whether the output result is correct, train the detection classifier.

[0046] After obtaining the neuron activation states corresponding to each training sample, the server can train a detection classifier that classifies the LLM output based on the neuron activation states. The training samples use the activation states of the training samples as input, and the labeling of the training samples indicates whether the output is correct. Therefore, this detection classifier, trained using these samples, does not concern itself with the LLM output or the LLM model input; it only needs to obtain the neuron activation states in the LLM to classify whether the neuron activation states are correct, essentially detecting whether "hallucinations" occur.

[0047] In the embodiments of this specification, the content obtained by the LLM after performing logical reasoning based on the model input is called the output result. The output result is divided into correct output result and incorrect output result. When the LLM exhibits "illusion," the LLM outputs an incorrect output result. Therefore, the output of this detection classifier is the classification of whether the LLM's output result is correct or incorrect.

[0048] Furthermore, the type of detection classifier is not limited in the embodiments of this specification. A classifier based on random forest can be selected, or different classifier model structures such as the optimized distributed gradient boosting library (eXtreme Gradient Boosting, XGBoost) and gradient boosting decision tree (GBDT) can be selected, and the specific configuration can be set as needed.

[0049] Of course, the server ultimately trains a detection classifier using the generated classification samples. This classifier is specifically designed for the LLM to be detected, because the activation states corresponding to the training samples in the classification samples are obtained through the LLM to be detected. This specification does not preclude the application of this detection classifier to other LLMs, but it is important to note that the neurons and model structure of other LLMs must be consistent with those of the LLM to be detected; otherwise, the accuracy of the detection classifier cannot be guaranteed. For example, a detection classifier trained for an LLM with 1000 neurons, if used for an LLM with 3000 neurons, will obviously struggle to provide reliable classification results.

[0050] S106: In response to the activation state of the neurons executing the business process of the large model to be detected, the correctness of the output result of the large model to be detected is detected by the detection classifier.

[0051] In the embodiments described in this specification, the detection classifier trained through the aforementioned steps can be used to detect the correctness of the LLM output results. Of course, the detection process is usually performed by a server, but this specification does not limit whether it is the server used to train the detection classifier or another server.

[0052] Furthermore, since it's necessary to obtain the activation states of neurons in the LLM to be tested, the server used to detect the correctness of the output result can be the same server as the server that executes the business through the LLM to be tested. That is, after the server executes the business using the LLM to be tested and determines the output result, it inputs the activation states of the neurons in the LLM to be tested into the detection classifier to determine whether the output result is correct. If the server performing the detection and the server executing the business are different devices, the server executing the business records the activation states of the neurons in the LLM to be tested and sends them to the server performing the detection. The server performing the detection uses the detection classifier to determine whether the output result is correct and then returns it to the server executing the business. The server executing the business determines whether to execute the business based on the feedback result. That is, when the output result is incorrect, the business is not executed based on that output result; when the output result is correct, the business is executed based on that output result.

[0053] Specifically, firstly, in response to the input data of the large model to be detected, the server can determine the activation state of neurons in the large model to be detected, which serves as the detection data, i.e., the input to the detection classifier. The process by which the server obtains the activation state of neurons is consistent with the process described in step S102, and therefore will not be repeated here.

[0054] Then, the server inputs the detection data into the trained detection classifier, and checks the correctness of the output of the large model to be detected based on the output of the detection classifier.

[0055] In the embodiments of this specification, if the LLM to be detected is used to execute business operations, then in the business scenario, the LLM to be detected is a business model. The server can then continue executing the business operations based on the output of the detection classifier, provided that the output of the business model is correct.

[0056] When the server determines that the output of the business model is incorrect based on the output of the detection classifier, and therefore determines that the business cannot be executed based on that output, the server determines the subsequent operations to be performed according to the business process and continues to execute the business. The specific business process specifies how to handle situations where the LLM output is incorrect, i.e., an "illusion" occurs; this specification does not impose restrictions and can be set as needed. For example, the user is asked to change their input text; based on the revised text, the output is obtained again through the LLM to be detected, and the detection is performed again.

[0057] It should be noted that in step S102, the preceding text only emphasized the role of erroneous output results and neuron activation states in training the detection classifier. However, on the other hand, the activation states of neurons corresponding to correct LLM output results are necessarily different from the activation states of neurons corresponding to erroneous LLM output results. Therefore, even if there are only a few or no training samples labeled as "hallucinations," they can still be used to determine whether the LLM output results are correct in subsequent training of the detection classifier. Essentially, the detection classifier trained in subsequent steps only needs to determine whether the neuron activation states correspond to the correct output results. Therefore, in this embodiment, the requirement for training samples labeled as "hallucinations" is not a necessary condition; however, all training samples are considered high-quality samples, and all output results contained within them are considered correct.

[0058] based on Figure 1 The large model output detection method described above identifies whether an LLM (Limited Least Meridian) is experiencing hallucinations by using training samples labeled with information on the correctness of the outputs and determining the activation states of neurons during the inference process. This method trains a detection classifier based on the relationship between the correctness of the output and the neuron activation states. When the LLM executes task input data, the activation states of neurons are captured, and the detection classifier is used to detect the correctness of the LLM's output, i.e., whether "hallucinations" are occurring. This method identifies whether an LLM is experiencing hallucinations by linking the activation states of LLM neurons to the occurrence of hallucinations, without needing to consider the LLM's inference process or output results; it identifies hallucinations at the model's underlying level.

[0059] Compared to existing methods for addressing model "illusion," such as manually removing spurious data from the training dataset—data that causes the model to "illusory"—to maintain the accuracy of LLM training, this approach relies on human experience and judgment, potentially leading to subjective biases and insufficient objectivity in data filtering. Furthermore, manually filtering the training dataset requires significant time and human resources. Rule-based methods for retrieving training datasets require expert-determined rules, which, besides being difficult to cover all types of spurious data, still require substantial manual effort, making it impossible to completely remove problematic spurious data from the training dataset. While SFT can significantly improve the ability of LLM to output correct results, it heavily depends on fine-tuning the LLM with a large amount of labeled, high-quality data. This type of data is difficult to obtain, and the fine-tuning process demands significant hardware resources.

[0060] However, the large model output detection method provided in this manual does not require retraining or fine-tuning the LLM, offering greater convenience. It also eliminates the need for manual removal of spurious data from the training dataset, saving significant manpower and time. Similarly, it does not rely on expert experience or rules, and the above steps can be implemented on different LLMs to achieve the same results. Compared to retrieval rules that may only apply to a single LLM, this method offers greater versatility.

[0061] In the embodiments of this specification, the training samples obtained in step S100 should match the LLM to be detected. Since the knowledge required for LLMs used in different business scenarios may differ, and the orientation of the output results may also differ, LLMs used in general business scenarios need to be fine-tuned through a corresponding SFT process. For example, assuming the business scenario is a legal consulting scenario, a general LLM is first obtained, and then, based on legal knowledge, training samples are determined to perform SFT on the LLM. In this specification, the LLM fine-tuned based on the business scenario is referred to as the business model.

[0062] Clearly, there are certain differences between the general LLM and the business model, and these differences are not limited to the differences in the activation states of neurons when faced with the same input. Therefore, in order to improve the accuracy of the trained detection classifier, in this specification, the server can also first train a filtering classifier for the general LLM to filter the training samples in step S100 from the general samples used to train the general LLM.

[0063] Specifically, the server first obtains general samples for training the LLM, which contain annotations indicating whether the model output is correct. Of course, the LLM used for training in these general samples is not the LLM to be detected, but a general LLM. Furthermore, as described in step S104, when the detection classifier is applied to other LLMs, it can be applied to other LLMs with the same structure as the LLM to be detected. Based on the same principle, when selecting general samples, general samples for training a general LLM with the same model structure as the LLM to be detected can also be selected based on the model structure of the LLM to be detected. Here, "same model structure" can mean two things: first, a consistent model architecture, such as the same number of neurons and neuron connections; and second, consistent neuron types. Of course, the latter requires more stringent conditions and can be used selectively. For example, first select general samples of general LLMs with consistent model architectures; if the number of general samples is sufficient, then further select general samples of general LLMs with consistent neuron types. The general samples of general LLMs mentioned in this specification refer to general samples used for training general LLMs.

[0064] Secondly, based on the input general sample, the activation state of neurons in the preset general large model is determined as the general state corresponding to the general sample;

[0065] Then, using the general state corresponding to the general sample as input and the labeling of whether the output result is correct, the classifier is trained and filtered.

[0066] The above two steps are similar to step S102, which inputs training samples into the LLM to be detected and determines the activation state, and step S104, which trains the detection classifier. The only difference is that the input data is transformed into general samples, the neurons of the general LLM are activated, and the activation states of the neurons of the general LLM are recorded (called the general state corresponding to the general sample). The training is for a selection classifier specific to the general LLM. Therefore, the relevant details can be found in the descriptions of steps S102 and S104, and will not be repeated here.

[0067] Finally, the server can select training samples from the general samples to train the detection classifier based on the second activation state of the neurons corresponding to the general samples in the large model to be detected, and the filtering classifier.

[0068] By leveraging the performance of general samples on the LLM to be tested, this filtering classifier selects general samples that still exhibit "hallucinations" from the general samples and uses them as training samples. In this way, even if there are no pre-labeled training samples indicating whether "hallucinations" occur in the business scenario corresponding to the LLM to be tested, general samples from the general scenario and the filtering classifier can be used to obtain samples that can still induce "hallucinations" in the LLM to be tested, which can then be used as training samples.

[0069] Specifically, to select training samples from the general samples, the first step is for the server to determine the activation state of neurons in the large model to be tested based on the input general samples, which is then used as the specific state corresponding to the general samples. In other words, the general samples are input into the LLM to be tested, and the activation states of neurons in the LLM to be tested during inference of the general samples are recorded and used as the specific states of the general samples.

[0070] The second step involves the server determining the correctness of the output results of the large model to be detected by filtering classifiers based on the specific states corresponding to the general samples, and determining the training samples based at least on the correctness of the output results.

[0071] In the embodiments described in this specification, the server takes a specific state as input to a filtering classifier to obtain a classification result indicating whether the output of the LLM to be detected is correct. That is, the filtering classifier determines whether the LLM to be detected is causing "illusion." If the output is determined to be incorrect, i.e., causing "illusion," then this general sample can be used as a training sample for training the detection classifier. Due to differences in business scenarios and model parameters, directly using general samples to train the detection classifier may lead to contamination of the detection classifier by inapplicable general samples. However, through the above filtering process, the obtained training samples can be determined to be samples that cause "illusion" in the LLM to be detected, thus avoiding contamination problems caused by inapplicable samples.

[0072] Furthermore, in the embodiments of this specification, the server can use only the general samples that cause "hallucinations" as training samples. Specifically, the server can determine the general samples whose output results are incorrect based on the correctness of the output results, and use the incorrect output results as labels to update the labels of whether the output results of the general samples are correct or not, thus obtaining training samples. In other words, for any general sample, regardless of whether the general sample causes "hallucinations" on the general LLM, as long as it causes "hallucinations" on the LLM to be detected, it can be used as a training sample to train the detection classifier.

[0073] Furthermore, in the embodiments of this specification, when determining training samples based on the correctness of the output results, the server may also determine samples from the general samples whose labeled output results are consistent with the output results obtained by the filtering classifier, as training samples. That is, the determined training samples are those that can induce "hallucinations" on both the general LLM and the LLM to be detected. This ensures the stability of the training samples, guaranteeing that the obtained training samples have sufficient ability to induce hallucinations in the LLM.

[0074] Of course, if the training samples are obtained by screening using the above method, then the step of determining the activation state of the training samples in step S102 can be replaced by the above process.

[0075] Figure 3 This is a schematic diagram of the large model output result detection process provided in the embodiments of this specification. Figure 3 To combine Figure 1 The steps and the process of obtaining training samples through the general sample selection described above are explained in detail above and will not be repeated here.

[0076] In the embodiments described in this specification, the server can also, through a human-computer interaction process, send general samples with incorrect output results to the user terminal based on the correctness of the output results. It can also determine training samples based on at least the labels returned by the user terminal.

[0077] Specifically, common samples that induce "hallucinations" on the LLM to be detected are sent to the user terminal, prompting the user to label whether the common sample can induce "hallucinations". The labeling of the common sample returned by the user terminal is manually labeled, and the common samples labeled as having incorrect output results are used as training samples.

[0078] Because the above filtering process significantly reduces the number of general samples that the server can send to user terminals, manual annotation can focus on annotating only general samples that the classifier considers to have incorrect output results. Illusory data typically constitutes a small proportion, greatly improving the efficiency of manual annotation.

[0079] The above is a method for detecting the output results of a large model provided in the embodiments of this specification. Based on the same idea, this specification also provides corresponding devices, storage media and electronic devices.

[0080] Figure 4 This is a schematic diagram of a large model output result detection device provided in an embodiment of this specification. The device includes:

[0081] The acquisition module 201 is used to acquire training samples for training a large model, wherein the training samples contain annotations indicating whether the model output results are correct;

[0082] Inference module 202 is used to determine the activation state of neurons in the large model to be detected based on the input training sample, and use it as the activation state corresponding to the training sample;

[0083] Training module 203 is used to train a detection classifier by taking the activation state corresponding to the training sample as input and labeling whether the output result is correct.

[0084] The detection module 204 is used to detect the correctness of the output result of the large model under test by means of the detection classifier in response to the activation state of the neurons executing the business process of the large model under test.

[0085] Optionally, the detection module 204 is configured to, in response to the input data of the large model to be detected, determine the activation state of neurons in the large model to be detected as detection data; input the detection data into a trained detection classifier; and, based on the output of the detection classifier, detect the correctness of the output result of the large model to be detected.

[0086] Optionally, the acquisition module 201 is configured to acquire general samples for training a large model, the general samples containing labels indicating whether the model output results are correct; determine the activation state of neurons in a preset general large model based on the input of the general samples, as the general state corresponding to the general samples; train a screening classifier using the general state corresponding to the general samples as input and the labels indicating whether the output results are correct; and select training samples for training the detection classifier from the general samples according to the activation state of neurons corresponding to the general samples in the large model to be detected and the screening classifier.

[0087] Optionally, the acquisition module 201 is used to determine the activation state of neurons in the large model to be detected based on the input general sample, as the specific state corresponding to the general sample; and to determine the correctness of the output result of the large model to be detected by the screening classifier according to the specific state, and to determine the training sample at least according to the correctness of the output result.

[0088] Optionally, the acquisition module 201 is used to update the label of whether the output result of the general sample is correct based on the correctness of the output result, so as to obtain the training sample.

[0089] Optionally, the acquisition module 201 is used to determine, from the general samples, samples whose labeled output results and the output results obtained by the filtering classifier are consistent, as training samples.

[0090] Optionally, the acquisition module 201 is used to send general samples with incorrect output results to the user terminal based on the correctness of the output results; and to determine training samples at least in response to the annotations returned by the user terminal.

[0091] Optionally, the large model to be detected is a business model;

[0092] The device further includes:

[0093] The execution module 205 is used to execute the business corresponding to the business model based on the output result when the output result of the business model is determined to be correct according to the output of the detection classifier.

[0094] This specification also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can be used to perform the large model output result detection method provided above.

[0095] based on Figure 1 The large model output result detection method shown in this specification is further illustrated in the embodiments. Figure 5 The diagram shows the structure of the electronic device. Figure 5 At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the aforementioned large model output result detection method.

[0096] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A method for detecting the output results of a large model, the method comprising: Training samples for training a large model are obtained. The training samples are text described in natural language. The training samples contain annotations indicating whether the model output is correct. The training samples are general samples that induce hallucinations on the large model to be tested, determined by a screening classifier based on the activation state of neurons in the general sample in the general large model. The general sample contains annotations indicating whether the model output is correct. The screening classifier is trained based on the activation state of neurons in the general sample in the general large model and the annotations of the general sample. Based on the input training samples, the activation state of neurons in the large model to be detected is determined as the activation state corresponding to the training samples. The activation state of the neurons includes whether each neuron is activated. Using the activation state corresponding to the training sample as input, and labeling whether the output result is correct, a detection classifier is trained. In response to the user inputting text of the large model to be detected when performing business, the activation state of the neurons when the large model to be detected obtains the output result is determined as detection data. The detection data is input into the detection classifier to determine the correctness of the output result. If the output result is determined to be correct, the business continues to be executed based on the output result. Otherwise, the user is prompted to modify the input text. The output result of the detection classifier controls whether to execute the business process.

2. The method as described in claim 1, wherein, in response to the activation state of neurons executing the business process of the large model to be detected, the correctness of the output result of the large model to be detected is detected by the detection classifier, specifically including: In response to the input data of the large model to be detected, the activation state of neurons in the large model to be detected is determined as detection data; The detection data is input into the trained detection classifier, and the correctness of the output result of the large model to be detected is detected based on the output of the detection classifier.

3. The method as described in claim 1, wherein obtaining training samples for training a large model specifically includes: Obtain general samples for training large models, the general samples containing labels indicating whether the model output is correct; Based on the input general sample, the activation state of neurons in a preset general large model is determined as the general state corresponding to the general sample; Using the general state corresponding to the general sample as input, and labeling whether the output result is correct, a screening classifier is trained; Based on the activation state of the corresponding neurons in the large model to be detected according to the general sample, and the screening classifier, training samples for training the detection classifier are selected from the general sample.

4. The method as described in claim 3, wherein training samples for training the detection classifier are selected from the general samples based on the activation state of the neurons corresponding to the general samples in the large model to be detected, and the screening classifier, specifically including: Based on the input general sample, the activation state of the neurons in the large model to be detected is determined as the specific state corresponding to the general sample; Based on the specific state, the correctness of the output result of the large model to be detected is determined by the filtering classifier, and training samples are determined at least based on the correctness of the output result.

5. The method as described in claim 4, wherein the training samples are determined based on the correctness of the output results, specifically including: Based on the correctness of the output results, update the label of whether the output results of the general sample are correct, and obtain the training samples.

6. The method as described in claim 4, wherein determining the training samples based on the correctness of the output results specifically includes: From the general samples, samples whose labeled output results are consistent with the output results obtained by the filtering classifier are selected as training samples.

7. The method as described in claim 4, wherein the training samples are determined based on the correctness of the output results, specifically including: Based on the correctness of the output results, general samples with incorrect output results are sent to the user terminal; At least in response to the annotations returned by the user terminal, training samples are determined.

8. The method as described in claim 2, wherein the large model to be detected is a business model; The method further includes: When the output of the detection classifier is determined to be correct, the business corresponding to the business model is executed based on the output.

9. A device for detecting the output results of a large model, the device comprising: The acquisition module is used to acquire training samples for training a large model. The training samples include annotations indicating whether the model output is correct. The training samples are text describing natural language. The training samples are general samples that induce hallucinations on the large model to be tested, determined by a screening classifier based on the activation state of neurons in the general sample in the general large model. The general sample includes annotations indicating whether the model output is correct. The screening classifier is trained based on the activation state of neurons in the general sample in the general large model and the annotations of the general sample. The inference module is used to determine the activation state of neurons in the large model to be detected based on the input training sample, as the activation state corresponding to the training sample, wherein the activation state of the neurons includes whether each neuron is activated. The training module is used to train a detection classifier by taking the activation state corresponding to the training sample as input and labeling whether the output result is correct. The detection module is used to respond to the text of the large model to be detected input by the user when performing business, determine the activation state of the neurons when the large model to be detected obtains the output result, use the detection data as detection data, input the detection classifier, determine the correctness of the output result, and when the output result is determined to be correct, continue to perform business based on the output result; otherwise, provide feedback to the user to request modification of the input text, and control whether to execute the business process based on the output result of the detection classifier.

10. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any one of claims 1-8.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method according to any one of claims 1-8.

Images