Model hallucination behavior positioning method and device, computer device, readable storage medium and program product

Abstract

The application relates to a model hallucination behavior positioning method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring a sample set, inputting samples in the sample set into a target model to obtain activation values of each sample on each unit in the target model; determining a weighted directed graph according to the activation values of each sample on each unit in the target model; determining a hallucination loop from the weighted directed graph through a search optimization strategy and a comprehensive score function; the comprehensive score function is used for evaluating the loop; performing an intervention operation on the hallucination loop, and determining the hallucination loop as a hallucination key loop in the case that a loop hallucination change amount and a loop non-hallucination retention rate both satisfy preset conditions. The method can accurately position the loop and the unit that generate hallucinations.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, computer device, computer-readable storage medium, and computer program product for locating model hallucination behavior. Background Technology

[0002] With the rapid development of cloud-native deployment, large model service-oriented architecture, and intelligent agent applications, large language models have been widely used for knowledge question answering, task planning, tool invocation, and industry decision support.

[0003] However, large language models are prone to generating seemingly reasonable but factually inaccurate content—a phenomenon known as "illusion"—when faced with unknown entities, insufficient evidence, conflicting contexts, or user manipulation. In cloud-native intelligent agent scenarios, illusions not only affect the accuracy of single-round responses but can also be amplified through task and tool chains, leading to higher business risks.

[0004] Traditional techniques mostly involve detection or correction at the output end, such as self-consistency, semantic entropy, and resampling verification. These methods can detect unreliable answers but are difficult to identify the hallucinogenic neurons. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, device, computer equipment, computer-readable storage medium, and computer program product for locating hallucinogenic neurons to address the above-mentioned technical problems.

[0006] Firstly, this application provides a method for localizing model hallucination behavior, the method comprising:

[0007] Obtain a sample set, input the samples in the sample set into the target model, and obtain the activation value of each sample in each unit of the target model;

[0008] A weighted directed graph is determined based on the activation values ​​of each sample in each unit of the target model;

[0009] Illusionary circuits are identified from the weighted directed graph using a search optimization strategy and a comprehensive scoring function; the comprehensive scoring function is used to evaluate the circuits.

[0010] An intervention operation is performed on the hallucination loop. If both the change in the hallucination amount and the retention rate of the non-hallucination of the loop meet preset conditions, the hallucination loop is identified as a key hallucination loop. The change in the hallucination amount is the change in the hallucination probability of the target model before and after the intervention operation. The retention rate of the non-hallucination of the loop is the retention rate of the correct answers of the target model after the intervention operation.

[0011] In one embodiment, the samples include hallucination samples and non-hallucination samples; the determination of the comprehensive scoring function includes:

[0012] Obtain loop scoring metrics and their corresponding weights; wherein, the loop scoring metrics include loop discriminability, loop illusion variableization, loop stability, number of loop edges, and number of loop nodes; the loop discriminability is the ability of a loop to distinguish between illusion samples and non-illusion samples; the loop stability is the stability of a loop in synonym rewriting, contextual perturbation, or multiple samples;

[0013] The comprehensive scoring function is determined based on the loop distinguishability, the illusion variableization, the loop stability, the number of loop edges and the number of loop nodes, and the corresponding weights.

[0014] In one embodiment, the method for determining the loop distinguishability includes:

[0015] Obtain the first support of the circuit in the hallucination sample and the second support in the non-hallucination sample;

[0016] The loop discrimination is determined based on the first support and the second support.

[0017] In one embodiment, determining the hallucinatory circuit as a key hallucinatory circuit when both the hallucinatory change rate and the non-hallucinatory retention rate of the circuit meet preset conditions includes:

[0018] If the change in the loop hallucination is greater than or equal to the change threshold, the non-hallucination retention rate is greater than or equal to the retention rate threshold, and the loop discrimination is greater than the discrimination threshold, the hallucination loop is identified as a key hallucination loop.

[0019] In one embodiment, determining the weighted directed graph based on the activation values ​​of each sample in each unit of the target model includes:

[0020] Based on the activation values ​​of each sample on each unit in the target model, a discriminative score is determined for each unit; wherein the samples include hallucination samples and non-hallucination samples; the discriminative score is used to evaluate the degree of difference in the distribution of activation values ​​of a unit on the hallucination samples and the non-hallucination samples;

[0021] Units with a discriminative score greater than the discriminative threshold are identified as candidate units;

[0022] The weighted directed graph is determined based on the directed dependencies between the candidate units.

[0023] In one embodiment, determining the discriminative score of each unit based on the activation values ​​of each sample in each unit of the target model includes:

[0024] Based on the activation values ​​of each hallucination sample and non-hallucination sample in each unit of the target model, determine the average activation value of different types of samples in the same unit;

[0025] Determine the corresponding standard deviation based on the average activation value of samples of different types in the same unit;

[0026] The discriminative score for each unit is determined based on the average activation value and the corresponding standard deviation of samples of different types in the same unit.

[0027] Secondly, this application also provides a model hallucination behavior localization device, the device comprising:

[0028] An input acquisition module is used to acquire a sample set, input the samples in the sample set into the target model, and obtain the activation value of each sample in each unit of the target model.

[0029] The first determining module is used to determine the weighted directed graph based on the activation values ​​of each sample in each unit of the target model;

[0030] The second determining module is used to determine the illusionary circuit from the weighted directed graph through a search optimization strategy and a comprehensive scoring function; the comprehensive scoring function is used to evaluate the circuit.

[0031] The third determining module is used to perform an intervention operation on the hallucination circuit. When both the hallucination change amount and the non-hallucination retention rate of the circuit meet preset conditions, the hallucination circuit is determined as a key hallucination circuit. The hallucination change amount is the change in the hallucination probability of the target model before and after the intervention operation; the non-hallucination retention rate of the circuit is the retention rate of the correct answer of the target model after the intervention operation.

[0032] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0033] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0034] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0035] The aforementioned method, apparatus, computer device, computer-readable storage medium, and computer program product for locating hallucination behavior first acquire a sample set, inputs the samples from the sample set into the target model, and obtains the activation value of each sample in each unit of the target model. Second, based on the activation value of each sample in each unit of the target model, a weighted directed graph is determined, providing a structural basis for subsequent discovery of "hallucination-related loops." Third, hallucination loops are identified from the weighted directed graph through a search optimization strategy and a comprehensive scoring function; the comprehensive scoring function is used to evaluate each loop. Finally, an intervention operation is performed on the hallucination loop. If both the change in the hallucination level and the retention rate of the non-hallucination level of the loop meet preset conditions, the hallucination loop is identified as a key hallucination loop. Here, the change in the hallucination level is the change in the hallucination probability of the target model before and after the intervention operation; the retention rate of the non-hallucination level of the loop is the retention rate of correct answers of the target model after the intervention operation. This method can accurately locate key hallucination loops that are "strongly correlated with hallucinations, effective after intervention, stable across perturbations, and structurally compact." Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a flowchart illustrating a method for locating model hallucination behavior in one embodiment;

[0038] Figure 2 This is a flowchart illustrating how the comprehensive scoring function is determined in one embodiment;

[0039] Figure 3 This is a flowchart illustrating the process of determining a weighted directed graph in one embodiment;

[0040] Figure 4 This is a flowchart illustrating the process of determining the discriminative score for each unit in one embodiment.

[0041] Figure 5 This is a structural block diagram of a model hallucination behavior localization device in one embodiment;

[0042] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0044] In one embodiment, such as Figure 1 As shown, a method for locating hallucination behavior in a model is provided. This embodiment illustrates the application of this method to a server. It is understood that this method can also be applied to a terminal, or to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps S102 to S108. Wherein:

[0045] Step S102: Obtain the sample set, input the samples in the sample set into the target model, and obtain the activation value of each sample in each unit of the target model.

[0046] The sample set can be a composite context sample set adapted to cloud-native intelligent agent scenarios.

[0047] Optionally, the server acquires a composite context sample set for cloud-native intelligent agent scenarios, enabling subsequent loop discovery to go beyond bare prompts and cover tool calls, context injection, and system constraints in real-world deployment scenarios. Acquiring the composite context sample set for cloud-native intelligent agent scenarios includes: acquiring ordinary question-and-answer samples and expanding them into a composite context sample set adapted to cloud-native intelligent agent scenarios. Expansion methods include, but are not limited to, multi-turn dialogue expansion, external knowledge injection context expansion, and tool call expansion.

[0048] The sample set is shown in formula (1).

[0049] Formula (1)

[0050] In the formula, Representing the One input question or task instruction; Represents contextual information, including but not limited to system prompts, search fragments, tool outputs, and agent intermediate states; Represents the model output; The label represents the hallucination, with 1 indicating a hallucination sample and 0 indicating a non-hallucination sample; N represents the total number of samples in the sample set.

[0051] Optionally, the large-scale diagnostic model is used to assist doctors in disease diagnosis and medication recommendations. Traditional models occasionally exhibit hallucinations when answering medical questions, such as misrepresenting drug interactions, fabricating clinical trial results, or confusing disease symptoms, potentially leading to medical risks. The server extracts question-answer pairs from medical textbooks, clinical guidelines, drug instructions, and real case databases, constructing two types of samples: hallucination samples and non-hallucination samples. Hallucination samples include examples where the model-generated answers contradict authoritative medical knowledge. Non-hallucination samples include model-generated answers consistent with authoritative medical knowledge and verified by the attending physician. Using both hallucination and non-hallucination samples from the medical field, the server calculates the activation values ​​of neurons in the large-scale diagnostic model, constructs a causal attribution graph, and identifies several key medical hallucination loops through a comprehensive scoring function. Based on these identified key hallucination loops, targeted parameter adjustments are made to the model, such as fine-tuning the weights of neurons in the loops or adding inhibitory gating at the loop's output, to suppress the hallucination behavior of the large-scale diagnostic model. Improving the reliability of the large-scale diagnostic model can be achieved by adjusting only a small number of parameters in the key hallucination loops.

[0052] Optionally, the legal intelligent consultation model is used to answer questions based on the user's input of a case description, providing legal provisions, precedents, or litigation suggestions. The server extracts question-and-answer pairs from a legal database, Supreme People's Court guiding cases, and legal documents, constructing two types of samples: illusion samples (e.g., incorrectly cited legal provisions, non-existent precedents, or inconsistent charges with criminal law provisions) and non-illusion samples (e.g., responses consistent with current laws and judicial interpretations, and verified by practicing lawyers). Using both illusion and non-illusion samples in the legal field, the server calculates the activation values ​​of each neuron in the legal intelligent consultation model, constructs a causal attribution graph, and identifies several key loops of legal illusion through a comprehensive scoring function. By suppressing and adjusting the parameters corresponding to these key loops in the legal intelligent consultation model, the reliability of the model output can be improved. When the server detects the activation of an illusionary key loop, it automatically triggers an alert and reverts to a safe response mode of "This question is beyond my knowledge; please consult a professional lawyer" to improve reliability.

[0053] Step S104: Determine the weighted directed graph based on the activation values ​​of each sample in each unit of the target model.

[0054] The target model is the model to be detected, such as a feedforward neural network, or other networks. Units include, but are not limited to, feedforward network hidden units (neurons), residual flow feature units, attention head units, or combinations thereof, such as the feature dimension at each token position in the residual flow, attention heads, etc. A neuron is a hidden unit of the feedforward network (FFN) in the model, i.e., each computational node in the FFN layer. For example, the activation value of each sample in each unit of the target model can be denoted as... ,in, This represents the model layer number; n represents the neuron or candidate feature number in that layer. Representative sample In unit Aggregate activation value on.

[0055] Optionally, the server calculates the activation value of each sample in each unit of the target model. The position of the unit is determined by the layer number of the model and the corresponding neuron or candidate feature number. The server filters the activation values ​​of each sample in each unit of the target model and analyzes the dependencies to determine a weighted directed graph.

[0056] By outputting sample-level attribution graphs, the "local computation process from input to output within the model" is represented as a computable, pruneable, and visualizeable graph structure, providing a structural basis for the subsequent discovery of "hallucination-related loops".

[0057] Step S106: Illusion circuits are determined from the weighted directed graph using a search optimization strategy and a comprehensive scoring function.

[0058] The comprehensive scoring function, determined based on loop discrimination, illusion variableization, loop stability, number of loop edges, and number of loop nodes, is used to evaluate the quality of a loop. A higher comprehensive score calculated by the comprehensive scoring function indicates a better loop, manifested in stronger discrimination ability and better stability.

[0059] Optionally, the server uses search optimization strategies, such as search, pruning, and merging strategies, to determine the circuits that satisfy the objective in the weighted directed graph. For example, the circuit with the highest comprehensive score or the top-K circuits are considered as "strongly correlated with hallucinations, effective after intervention, stable across perturbations, and with a sufficiently compact structure" hallucination circuits. Where K is a positive integer.

[0060] Step S108: Perform an intervention operation on the hallucination circuit. If both the hallucination change rate and the non-hallucination retention rate of the circuit meet the preset conditions, the hallucination circuit is identified as a key hallucination circuit.

[0061] Wherein, the change in loop illusion is denoted as It is the change in the probability of hallucination of the target model before and after the intervention operation, and its calculation method is shown in formula (2).

[0062] Formula (2)

[0063] In the formula, Representative sample Hallucination probability or hallucination score before intervention. Represents the loop The probability or score of hallucination after suppression. M represents the number of samples involved.

[0064] To prevent "severely impairing normal response ability while eliminating hallucinations," the non-hallucination retention rate is defined as . That is, the loop non-hallucination retention rate is the retention rate of the correct answer of the target model after the intervention operation, as shown in formula (3).

[0065] Formula (3)

[0066] In the formula, Represents a set of non-hallucination samples; Represents an indicator function; This represents the output after the intervention operation.

[0067] Optionally, the server supports the illusion circuit. The system performs intervention operations such as suppression, amplification, zeroing, replacement, or edge pruning, calculating the change in the hallucination probability of the target model before and after the intervention. To prevent "severely impairing normal response ability while eliminating hallucinations," a non-hallucination retention rate is defined as the retention rate of correct responses of the target model after the intervention. The server calculates the retention rate of correct responses of the target model after the intervention. If both the change in loop hallucination and the retention rate of loop non-hallucination meet preset conditions, the hallucination loop is identified as a key hallucination loop. The server outputs structured explanation information of the key hallucination loop, such as a topology graph, node semantic labels, and triggering conditions, and stores it in the loop knowledge base.

[0068] Furthermore, when the target model, such as a large language model, is actually deployed, multiple processors can detect a small number of neurons corresponding to the hallucination critical loop in parallel, eliminating the need to monitor hundreds of billions of parameters in the entire model, thus significantly reducing the computational load. The nodes in the hallucination critical loop are decomposed into several independent subtasks by layer or by dependency. For example, neurons within the same layer are grouped together, and neurons between different layers can be decomposed into pipeline stages due to the order of forward propagation. The server assigns each subtask to different processors for parallel execution. Each processor only needs to calculate the activation values ​​of the few neurons assigned to it and synchronize some intermediate results (such as cross-layer dependency nodes) with other processors. Since the total number of nodes in the hallucination critical loop is small, the computational load of each subtask is minimal, and the communication overhead between processors is far lower than that of scanning the entire model. Simultaneously, the topology, units (which layer, which neuron), edge weights, trigger thresholds, etc., of the hallucination critical loop are pre-stored in a lightweight monitoring configuration file. Based on the configuration file, the previously calculated activation values ​​of neurons belonging to the hallucination critical loop are copied, and the activation values ​​of the hallucination critical loop are calculated. This is far less computationally intensive than calculating the activation values ​​of all loops in the model, significantly reducing memory usage.

[0069] In the aforementioned method for locating hallucination behavior, firstly, a sample set is acquired, and the samples in the sample set are input into the target model to obtain the activation values ​​of each sample in each unit of the target model. Secondly, based on the activation values ​​of each sample in each unit of the target model, a weighted directed graph is determined, providing a structural basis for the subsequent discovery of "hallucination-related loops". Thirdly, hallucination loops are identified from the weighted directed graph through a search optimization strategy and a comprehensive scoring function; the comprehensive scoring function is used to evaluate each loop. Finally, intervention operations are performed on the hallucination loops. If both the change in the hallucination level and the retention rate of the non-hallucination level of the loop meet preset conditions, the hallucination loop is identified as a key hallucination loop. Here, the change in the hallucination level of the loop is the change in the hallucination probability of the target model before and after the intervention operation; the retention rate of the non-hallucination level of the loop is the retention rate of the correct answers of the target model after the intervention operation, which can accurately locate key hallucination loops that are "strongly correlated with hallucination, effective after intervention, stable across perturbations, and have a sufficiently compact structure".

[0070] In one exemplary embodiment, such as Figure 2 As shown, the samples include hallucination samples and non-hallucination samples; the determination of the comprehensive scoring function includes the following steps S202 to S204. Wherein:

[0071] Step S202: Obtain the loop scoring index and its corresponding weight.

[0072] Among them, the loop scoring index includes loop discrimination, denoted as The variableization of loop hallucinations, that is, the strength of the influence on the probability of hallucination after intervening in the loop, is denoted as . Loop stability, denoted as The number of loop edges is denoted as The number of loop nodes is denoted as Loop discriminability is the ability of a loop to distinguish between hallucinatory and non-hallucinatory samples; loop stability is the stability of a loop in paraphrasing, contextual perturbation, or multiple samples.

[0073] The discrimination weight corresponding to the loop discrimination is denoted as: The value is used to control the contribution of the loop's discriminative power between hallucination and non-hallucination samples to the overall score. The larger the value, the more important the loop's "discriminative power," meaning the server is more inclined to find loops that can clearly distinguish between hallucination and non-hallucination.

[0074] The causal effect weights corresponding to the variableization of loop illusion are denoted as follows: This value measures the weight of the contribution of intervening in this loop to the change in the probability of hallucination in the model's output in the overall score. The larger the value, the more the server tends to select loops that have a strong causal effect on the hallucination output.

[0075] The stability weight corresponding to the loop stability is denoted as This value is used to evaluate the contribution of the circuit's stability under paraphrasing, contextual perturbation, or multiple samples to the score. A higher value indicates that the server is more inclined to select circuits that are stable under different sample or perturbation conditions, ensuring reproducibility and generalization ability.

[0076] The edge penalty weight corresponding to the number of loop edges is denoted as: This is used to penalize the complexity of the number of edges in a cycle, avoiding overly bloated cycles. The larger the value, the more the scoring function will tend to select cycles that are more compact and have shorter computational paths.

[0077] The node penalty weight corresponding to the number of loop nodes is denoted as: This is used to penalize the number of nodes in a loop, controlling the loop size. The larger the value, the more the server tends to select loops with fewer nodes, thus maintaining the loop's simplicity and visual friendliness.

[0078] The servers obtain the loop discrimination scores respectively. and the corresponding discrimination weights ;Circuit illusion variableization Corresponding causal effect weight ; loop stability Corresponding stability weights Number of loop edges Corresponding edge penalty weight Number of loop nodes Corresponding node penalty weight .

[0079] Step S204: Determine the comprehensive scoring function based on loop discrimination, illusion variableization, loop stability, number of loop edges and number of loop nodes, and their corresponding weights.

[0080] Optionally, to ensure that the findings simultaneously possess "illusion discrimination, causal validity, cross-sample stability, and structural compactness," the server uses loop discrimination. and the corresponding discrimination weights ;Circuit illusion variableization Corresponding causal effect weight ; loop stability Corresponding stability weights Number of loop edges Corresponding edge penalty weight Number of loop nodes Corresponding node penalty weight Determine the comprehensive scoring function As shown in formula (4).

[0081] Formula (4)

[0082] As can be seen from formula (4), the comprehensive scoring function is essentially a comprehensive score with a penalty term. Loop discrimination, loop illusion variableization, and loop stability are positive indicators, while the number of loop edges and the number of loop nodes are negative indicators. Loop discrimination The bigger the better; loop illusion variableization The larger the better; loop stability The larger the better; number of loop edges The smaller the better; number of loop nodes The smaller the better. By adjusting the weights of each element, the loop with the best overall performance can be found.

[0083] In this embodiment, a comprehensive scoring function is constructed using the loop scoring index, which can determine the hallucination loops in terms of hallucination discriminability, causal validity, cross-sample stability, and structural compactness.

[0084] In an exemplary embodiment, the method for determining the loop discrimination score includes: obtaining a first support score of the loop in hallucination samples and a second support score in non-hallucination samples; and determining the loop discrimination score based on the first support score and the second support score.

[0085] Support measures the degree to which loop C is activated in a sample set. Support includes first support and second support.

[0086] The server can calculate support using a binarization method. For each sample, it determines whether all nodes in loop C are active (e.g., the activation value exceeds a certain threshold). If so, the support of that sample for loop C is 1; otherwise, it is 0. The first support is then calculated. =The proportion of samples in the hallucination sample where the circuit is activated.

[0087] The server can also calculate support using continuous value methods. For each sample, it calculates the overall activation strength of loop C, for example, by aggregating the activation values ​​of all nodes according to a certain rule (weighted sum, geometric mean, etc.), and then dividing by the maximum possible value to obtain a score between 0 and 1. =The average of the scores across all hallucination samples.

[0088] Optionally, the server obtains the loop. The first support in the hallucination sample is denoted as . And the second support in non-hallucination samples and negative sample penalty coefficient The negative sample penalty coefficient It is an adjustable hyperparameter used to control the penalty strength for loop activation in non-hallucination samples. Based on the first support. And second support Negative sample penalty coefficient The loop discrimination is determined as shown in formula (5).

[0089] Formula (5)

[0090] It can be seen from formula (5) that when When >0, It will be multiplied by Later from Subtract from the middle. In other words, if the loop is frequently activated in non-hallucination samples, its discrimination score will be reduced. Therefore, The larger the value, the heavier the penalty for "circuit appearing in non-hallucination samples", and the server will tend to select circuits that appear almost exclusively in hazy samples.

[0091] In this embodiment, the loop discrimination is determined by support and negative sample penalty coefficient, which can screen out loops that are both statistically and causally related, ensuring that the loops are statistically specific and sensitive to hallucination samples and avoiding spuriously related loops.

[0092] In an exemplary embodiment, when both the loop hallucination change and the loop non-hallucination retention rate meet preset conditions, the hallucination loop is identified as a hallucination critical loop, including: when the loop hallucination change is greater than or equal to a change threshold, the non-hallucination retention rate is greater than or equal to a retention rate threshold, and the loop discrimination is greater than a discrimination threshold, the hallucination loop is identified as a hallucination critical loop.

[0093] Wherein, the threshold of change is denoted as The retention rate threshold is denoted as The discrimination threshold is denoted as .

[0094] Optionally, in In the case of loop illusion, that is, the change in the amount of loop illusion. Greater than or equal to the change threshold Non-hallucination retention rate Greater than or equal to the retention rate threshold Circuit discrimination Greater than the discrimination threshold The server identifies the hallucination circuit as the critical circuit of the hallucination.

[0095] In this embodiment, by filtering the conditions, the key circuit of hallucination can be accurately selected.

[0096] In one exemplary embodiment, such as Figure 3 As shown, a weighted directed graph is determined based on the activation values ​​of each sample in each unit of the target model, including steps S302 to S306. Wherein:

[0097] Step S302: Determine the discriminative score of each unit based on the activation value of each sample in each unit of the target model.

[0098] The samples include hallucination samples and non-hallucination samples; the discriminative score is used to assess the degree of difference in the distribution of activation values ​​of a unit on hallucination samples and non-hallucination samples.

[0099] Optionally, the server determines the activation value of each sample on each unit in the target model. The server determines the discriminative score for each unit. For example, the discriminative score of the third neuron in the second layer of the target model. The server calculates the discriminative score for each unit based on its activation value. .

[0100] Step S304: Units with a discrimination score greater than the discrimination threshold are identified as candidate units.

[0101] Wherein, the discrimination threshold is denoted as .

[0102] Optionally, the server selects the one that satisfies... Alternatively, units that achieve a Top-K score based on their discriminative scores can be used as candidate units, and these candidate units can form a set of candidate nodes. .

[0103] Step S306: Determine the weighted directed graph based on the directed dependencies between candidate units.

[0104] Optionally, the server targets a set of candidate nodes. For candidate units in the dataset, estimate the influence strength between candidate units (based on activation replacement, path repair, local perturbation, or equivalent causal attribution methods), and construct a weighted directed graph, denoted as . ;in, Representative sample The set of candidate nodes; The set of directed edges representing candidate units; Represents the set of edge weights.

[0105] either side The influence weights are shown in formula (6).

[0106] Formula (6)

[0107] In the formula: Representative sample Middle node Activation; Representative node A measure of the downstream output; Representative of the perturbation node After node Characterizing the amount of change; This represents the number of samples involved in the estimation.

[0108] In this embodiment, a sample-level directed graph is constructed using candidate units determined by discriminative scoring. This graph can represent the "local computation process from input to output within the model" as a computable, pruneable, and visualizeable graph structure, providing a structural basis for the subsequent discovery of hallucination circuits.

[0109] In one exemplary embodiment, such as Figure 4 As shown, the discriminative score for each unit is determined based on the activation values ​​of each sample in each unit of the target model, including steps S402 to S406. Wherein:

[0110] Step S402: Determine the average activation value of different types of samples in the same unit based on the activation values ​​of each hallucination sample and non-hallucination sample in each unit of the target model.

[0111] Optionally, the server determines the units in the target model based on each hallucination and non-hallucination sample. The activation value is used to determine the average activation value of different types of samples in the same unit. For example, the activation value of all hallucination samples in each unit and the activation value of all non-hallucination samples in each unit are calculated. The server calculates the average activation value of the hallucination samples in each unit based on the activation values ​​of all hallucination samples in each unit, denoted as . Similarly, the server calculates the average activation value of the non-hallucinatory samples in each unit based on the activation values ​​of all non-hallucinatory samples in each unit, denoted as . .

[0112] Step S404: Determine the corresponding standard deviation based on the average activation value of samples of different types in the same unit.

[0113] Optionally, the server determines the corresponding standard deviation based on the average activation value of samples of different types in the same unit, that is, based on the average activation value of the hallucination samples in each unit. Determine the corresponding standard deviation Based on the average activation value of non-hallucination samples in each unit Determine the corresponding standard deviation .

[0114] Step S406: Determine the discriminative score for each unit based on the average activation value and corresponding standard deviation of samples of different types in the same unit.

[0115] Optionally, by using the average activation value of samples of different types in the same unit ( and ) and the corresponding standard deviation ( and The ratio of ) is used to determine the discrimination score for each unit. As shown in formula (7),

[0116] Formula (7)

[0117] The greater the difference between the average activation value of a unit on hallucination samples and the average activation value on non-hallucination samples, the higher the discriminative score; the greater the fluctuation (standard deviation) of the activation value of a unit on any type of sample, the lower the discriminative score; the discriminative score reflects the unit's ability to distinguish hallucinations, that is, the degree to which it can effectively separate hallucination samples from non-hallucination samples. It is a minimal constant that prevents the denominator from being zero.

[0118] In this embodiment, the units are screened by discriminative scoring, compressing the original massive neuron space into candidate units that are "most significantly different from hallucination manifestations", thereby reducing the complexity of subsequent circuit search.

[0119] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0120] Based on the same inventive concept, this application also provides a model hallucination behavior localization device for implementing the model hallucination behavior localization method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more model hallucination behavior localization device embodiments provided below can be found in the limitations of the model hallucination behavior localization method described above, and will not be repeated here.

[0121] In one exemplary embodiment, such as Figure 5 As shown, a device for locating model hallucination behavior is provided, comprising: an input acquisition module 501, a first determination module 502, a second determination module 503, and a third determination module 504, wherein:

[0122] The input acquisition module 501 is used to acquire a sample set, input the samples in the sample set into the target model, and obtain the activation value of each sample in each unit of the target model.

[0123] The first determining module 502 is used to determine the weighted directed graph based on the activation values ​​of each sample in each unit of the target model.

[0124] The second determination module 503 is used to determine the illusionary circuit from the weighted directed graph through a search optimization strategy and a comprehensive scoring function; the comprehensive scoring function is used to evaluate the circuit.

[0125] The third determination module 504 is used to perform intervention operations on the hallucination loop. When both the change in the hallucination amount and the retention rate of the non-hallucination of the loop meet preset conditions, the hallucination loop is determined as the key hallucination loop. The change in the hallucination amount is the change in the hallucination probability of the target model before and after the intervention operation; the retention rate of the non-hallucination of the loop is the retention rate of the correct answer of the target model after the intervention operation.

[0126] In an exemplary embodiment, the samples include hallucination samples and non-hallucination samples; the model hallucination behavior localization device further includes: a comprehensive scoring function determination module, used to obtain loop scoring indicators and corresponding weights; wherein, the loop scoring indicators include loop discrimination, loop hallucination variableization, loop stability, number of loop edges, and number of loop nodes; loop discrimination is the degree to which a loop distinguishes between hallucination samples and non-hallucination samples; loop stability is the stability of a loop in synonym rewriting, contextual perturbation, or multiple samples; the comprehensive scoring function is determined based on the loop discrimination, hallucination variableization, loop stability, number of loop edges, number of loop nodes, and corresponding weights.

[0127] In an exemplary embodiment, the loop discrimination determination module is used to obtain a first support of the loop in hallucination samples and a second support in non-hallucination samples; and to determine the loop discrimination based on the first support and the second support.

[0128] In an exemplary embodiment, the third determining module 504 is further configured to determine the hallucination circuit as a key hallucination circuit when the change in the loop hallucination is greater than or equal to a change threshold, the non-hallucination retention rate is greater than or equal to a retention rate threshold, and the circuit discrimination is greater than a discrimination threshold.

[0129] In an exemplary embodiment, the first determining module 502 is further configured to determine the discriminative score of each unit based on the activation values ​​of each sample on each unit in the target model; wherein the samples include hallucination samples and non-hallucination samples; the discriminative score is used to evaluate the degree of difference in the distribution of activation values ​​of a unit on hallucination samples and non-hallucination samples; units with discriminative scores greater than the discriminative threshold are determined as candidate units; and a weighted directed graph is determined based on the directed dependencies between candidate units.

[0130] In an exemplary embodiment, the first determining module 502 is further configured to determine the average activation value of different types of samples in the same unit based on the activation values ​​of each hallucination sample and non-hallucination sample in each unit of the target model; determine the corresponding standard deviation based on the average activation value of different types of samples in the same unit; and determine the discriminative score of each unit based on the average activation value and the corresponding standard deviation of different types of samples in the same unit.

[0131] Each module in the aforementioned hallucination behavior localization device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0132] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores hallucination behavior data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a method for locating model hallucination behaviors.

[0133] Those skilled in the art will understand that Figure 6The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0134] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0135] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0136] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0137] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0138] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0139] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for localizing hallucination behavior in a model, characterized in that, The method includes: Obtain a sample set, input the samples in the sample set into the target model, and obtain the activation value of each sample in each unit of the target model; A weighted directed graph is determined based on the activation values ​​of each sample in each unit of the target model; Illusionary circuits are identified from the weighted directed graph using a search optimization strategy and a comprehensive scoring function; the comprehensive scoring function is used to evaluate the circuits. An intervention operation is performed on the hallucination loop. If both the change in the hallucination amount and the retention rate of the non-hallucination of the loop meet preset conditions, the hallucination loop is identified as a key hallucination loop. The change in the hallucination amount is the change in the hallucination probability of the target model before and after the intervention operation. The retention rate of the non-hallucination of the loop is the retention rate of the correct answers of the target model after the intervention operation.

2. The method according to claim 1, characterized in that, The samples include hallucination samples and non-hallucination samples; the method for determining the comprehensive scoring function includes: Obtain loop scoring metrics and their corresponding weights; wherein, the loop scoring metrics include loop discriminability, loop illusion variableization, loop stability, number of loop edges, and number of loop nodes; the loop discriminability is the ability of a loop to distinguish between illusion samples and non-illusion samples; the loop stability is the stability of a loop in synonym rewriting, contextual perturbation, or multiple samples; The comprehensive scoring function is determined based on the loop distinguishability, the illusion variableization, the loop stability, the number of loop edges and the number of loop nodes, and the corresponding weights.

3. The method according to claim 2, characterized in that, The method for determining the loop discrimination includes: Obtain the first support of the circuit in the hallucination sample and the second support in the non-hallucination sample; The loop discrimination is determined based on the first support and the second support.

4. The method according to claim 2 or 3, characterized in that, The step of identifying the hallucinatory circuit as a key hallucinatory circuit when both the hallucinatory change rate and the non-hallucinatory retention rate of the circuit meet preset conditions includes: If the change in the loop hallucination is greater than or equal to the change threshold, the non-hallucination retention rate is greater than or equal to the retention rate threshold, and the loop discrimination is greater than the discrimination threshold, the hallucination loop is identified as a key hallucination loop.

5. The method according to claim 1, characterized in that, The step of determining the weighted directed graph based on the activation values ​​of each sample in each unit of the target model includes: Based on the activation values ​​of each sample on each unit in the target model, a discriminative score is determined for each unit; wherein the samples include hallucination samples and non-hallucination samples; the discriminative score is used to evaluate the degree of difference in the distribution of activation values ​​of a unit on the hallucination samples and the non-hallucination samples; Units with a discriminative score greater than the discriminative threshold are identified as candidate units; The weighted directed graph is determined based on the directed dependencies between the candidate units.

6. The method according to claim 5, characterized in that, The step of determining the discriminative score for each unit based on the activation values ​​of each sample in each unit of the target model includes: Based on the activation values ​​of each hallucination sample and non-hallucination sample in each unit of the target model, determine the average activation value of different types of samples in the same unit; Determine the corresponding standard deviation based on the average activation value of samples of different types in the same unit; The discriminative score for each unit is determined based on the average activation value and the corresponding standard deviation of samples of different types in the same unit.

7. A device for locating model hallucination behavior, characterized in that, The device includes: An input acquisition module is used to acquire a sample set, input the samples in the sample set into the target model, and obtain the activation value of each sample in each unit of the target model. The first determining module is used to determine the weighted directed graph based on the activation values ​​of each sample in each unit of the target model; The second determining module is used to determine the illusionary circuit from the weighted directed graph through a search optimization strategy and a comprehensive scoring function; the comprehensive scoring function is used to evaluate the circuit. The third determining module is used to perform an intervention operation on the hallucination circuit. When both the hallucination change amount and the non-hallucination retention rate of the circuit meet preset conditions, the hallucination circuit is determined as a key hallucination circuit. The hallucination change amount is the change in the hallucination probability of the target model before and after the intervention operation; the non-hallucination retention rate of the circuit is the retention rate of the correct answer of the target model after the intervention operation.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images