Method and apparatus for predicting category of output information, storage medium, and electronic device

Abstract

This application discloses a method, apparatus, storage medium, and electronic device for predicting the category of output information. The method includes: when the target model completes the forward propagation of the input information and has not generated the first output lexical corresponding to the input information, obtaining the first content tendency value corresponding to each of the processing levels within the target model, wherein the first content tendency value is a scalar value obtained by projection transformation of the hidden state vector of each processing level; arranging multiple first content tendency values ​​based on the processing depth of each processing level to generate a numerical sequence for characterizing the change of the first content tendency value with the processing depth; performing a convergence operation on the numerical sequence to obtain a comprehensive representative value, and inputting the comprehensive representative value into a preset classifier; predicting the category of the output information that the target model will generate based on the preset classifier, wherein the category of the output information includes: factual output information and non-factual output information.

Description

Technical Field

[0001] This application relates to the field of large language model technology, and more specifically, to a method and apparatus for predicting the category of output information, a storage medium, and an electronic device. Background Technology

[0002] Large Language Models (LLMs) exhibit a phenomenon called "illusion" in natural language generation tasks, where generated content appears fluent but does not reflect reality. This problem persists even after LLMs have undergone alignment training. Research indicates that the internal representations of LLMs contain linear, separable signals of truth, theoretically providing a basis for determining the authenticity of their own output.

[0003] Most existing white-box methods are based on the analysis of hidden states within a specific layer of an LLM, which has many shortcomings. For example, while contrastive consistency search can discover potential truth directions under unsupervised conditions, it requires paired positive and negative statements and can only read representation information at a single depth. Intervention methods during inference implicitly assume that a specific layer is the optimal location of the truth signal and achieve the goal by manipulating the model's behavior at a specific layer. When training a linear classifier on a single layer of hidden states, the selection of the optimal reading layer is highly dependent on the specific dataset, and it usually only classifies a single terminal hidden state, thus limiting its generalization ability.

[0004] In related technologies, technical detection schemes for large language models (LLMs) treat the realism of an LLM as a static snapshot, assuming that the realism signal is fixed on a specific layer of the network. This method discards the discriminative information distributed across dozens of Transformer blocks, forcibly collapsing the deep, progressive realism construction process of the model into a single-point estimate. Mechanistically, the feedforward layer, as a distributed key-value memory, refines the prediction layer by layer, and the residual stream, as a representation transmitted and combined by a shared bus, means that realism is actually constructed gradually through layer-by-layer propagation, rather than being encoded instantaneously at a single layer. Single-layer reading, because it only glimpses a local section of this dynamic process, fails to capture complementary class differences in cross-layer drift; is susceptible to interference from single-layer noise and layer-specific perturbations; and loses the opportunity to utilize deep collaboration to suppress irrelevant fluctuations and enhance shared class signals.

[0005] Therefore, the relevant technologies suffer from the problem that the static state of a large model, which is only read from a single processing depth, makes it impossible to accurately determine the authenticity of the output content of the large model. Summary of the Invention

[0006] This application provides a method, apparatus, storage medium, and electronic device for predicting the category of output information, in order to at least solve the problem in related technologies that the inability to accurately determine the authenticity of the output content of a large model is caused by only reading the static state of a single processing depth of a large model.

[0007] According to one embodiment of this application, a method for predicting the category of output information is provided, comprising: when a target model completes forward propagation of input information and has not generated the first output lexical unit corresponding to the input information, obtaining first content tendency values ​​corresponding to all processing levels within the target model, wherein the first content tendency value is a scalar value obtained by projection transformation of the hidden state vector of each processing level; arranging multiple first content tendency values ​​based on the processing depth of each processing level to generate a numerical sequence for characterizing the change of the first content tendency value with processing depth; performing a convergence operation on the numerical sequence to obtain a comprehensive representative value, and inputting the comprehensive representative value into a preset classifier; predicting the category of output information to be generated by the target model according to the preset classifier, wherein the category of output information includes: factual output information and non-factual output information.

[0008] In an exemplary embodiment, obtaining the first content tendency value corresponding to each of the processing layers within the target model includes: obtaining a first hidden state vector at the output of the attention module of each processing layer or at the output of each processing layer, wherein the first hidden state vector is the hidden state vector generated by the target model at each processing layer at the time step corresponding to the last word of the input information; and performing a projection transformation on each first hidden state vector based on a preset projection rule corresponding to each processing layer to obtain the first content tendency value corresponding to each processing layer.

[0009] In an exemplary embodiment, obtaining a first hidden state vector at the output of the attention module of each processing level or at the output of each processing level includes: determining a target attention head for each processing level, wherein the attention module of each processing level includes the target attention head; obtaining a first output vector of the target attention head at the time step corresponding to the last lexical of the input information, and determining the first output vector as the first hidden state vector of each processing level.

[0010] In an exemplary embodiment, obtaining a first hidden state vector at the output of the attention module of each processing level or at the output of each processing level further includes: obtaining a second output vector of at least two attention heads of each processing level at the time step corresponding to the last word of the input information, wherein the attention module of each processing level includes at least two attention heads of each processing level; concatenating or weighting the second output vectors of the at least two attention heads to obtain an aggregate vector, and determining the aggregate vector as the first hidden state vector of each processing level.

[0011] In an exemplary embodiment, obtaining a first hidden state vector at the output of the attention module of each processing layer or at the output of each processing layer further includes: obtaining a third output vector output by each processing layer through the output after residual connection and normalization operations, and determining the third output vector as the first hidden state vector of each processing layer.

[0012] In an exemplary embodiment, before projecting each first hidden state vector based on a preset projection rule corresponding to each processing level, the method further includes: constructing a set of sample pairs containing labels, wherein each sample pair in the set contains a sample input information and a sample output information, each sample pair corresponds to a label, the label being used to indicate whether the sample output information in each sample pair is factual output information or non-factual output information; inputting the sample input information in each sample pair into the target model after freezing parameters to determine a second hidden state vector during the processing of the sample input information by each processing level, wherein the second hidden state vector and the first hidden state vector are taken from the same processing position and the same word position, and the second hidden state vector and the first hidden state vector have the same dimension; constructing the projection of each processing level. The vector is used to generate a second content bias value corresponding to each processing level by projecting the projection vector of each processing level onto each second hidden state vector. The dimension of the projection vector is the same as the dimension of the second hidden state vector. A loss function is constructed for each projection vector based on the second content bias value corresponding to each processing level and the label of each sample pair. Based on each loss function, the projection vector of each processing level is updated by backpropagation algorithm until a preset termination condition is met. A preset projection rule for each processing level is determined according to each updated projection vector. The preset termination condition includes at least one of the following: the number of updates of the projection vector reaches a preset number; the sum of the loss function values ​​corresponding to all processing levels is less than a preset loss threshold; and the norm change of the projection vectors corresponding to all processing levels is less than a preset change threshold in two adjacent iterations of the projection vector update.

[0013] In an exemplary embodiment, projecting a transformation on each second hidden state vector based on the projection vector of each processing level to generate a second content tendency value corresponding to each processing level includes: performing a dot product operation between the projection vector of each processing level and the second hidden state vector of each processing level to obtain a second content tendency value in scalar form.

[0014] In one exemplary embodiment, the loss function may be of one of the following types: binary cross-entropy loss function or contrastive loss function.

[0015] In one exemplary embodiment, the preset classifier is a logistic regression classifier.

[0016] In one exemplary embodiment, performing a convergence operation on the numerical sequence to obtain a comprehensive representative value includes: calculating the sum of a plurality of first content tendency values ​​contained in the numerical sequence; calculating the arithmetic mean of the plurality of first content tendency values ​​contained in the numerical sequence based on the sum of the plurality of first content tendency values ​​and the number of first content tendency values ​​contained in the numerical sequence, and determining the arithmetic mean as the comprehensive representative value.

[0017] In one exemplary embodiment, performing a convergence operation on the numerical sequence to obtain a comprehensive representative value further includes: determining the median among a plurality of first content tendency values ​​contained in the numerical sequence, and determining the median as the comprehensive representative value.

[0018] In one exemplary embodiment, performing a convergence operation on the numerical sequence to obtain a comprehensive representative value further includes: determining the weight corresponding to each first content tendency value contained in the numerical sequence; and performing a weighted average of the multiple first content tendency values ​​contained in the numerical sequence according to the weights to obtain the comprehensive representative value.

[0019] In an exemplary embodiment, performing a convergence operation on the numerical sequence to obtain a comprehensive representative value further includes: extracting at least one trajectory summarization parameter from the numerical sequence, wherein the trajectory summarization parameter includes at least one of the following: a dispersion parameter characterizing the fluctuation amplitude of the numerical sequence, a trend parameter characterizing the changing trend of the numerical sequence, and an endpoint difference parameter characterizing the difference between the initial state and the final state of the numerical sequence; combining the at least one trajectory summarization parameter into a target feature vector, and mapping the target feature vector to the comprehensive representative value.

[0020] In an exemplary embodiment, after predicting the category of the output information to be generated by the target model according to the preset classifier, the method further includes: if it is determined that the output information is non-factual output information, sending a stop instruction to the target model to cause the target model to stop generating the output information; obtaining a guide word corresponding to the input information, and generating target input information according to the guide word and the input information; inputting the target input information into the target model; sending a generation instruction to the target model to cause the target model to generate target output information according to the guide word and the input information, and determining the category of the target output information before the target model generates the target output information, wherein the category of the target output information includes: factual output information and non-factual output information; if it is determined that the target output information is factual output information, instructing the target model to generate the target output information.

[0021] In one exemplary embodiment, the guiding word is obtained by at least one of the following methods: obtaining the guiding word from a target knowledge base; obtaining the guiding word from a preset fact template; or obtaining the guiding word from a historical factual output information database.

[0022] In an exemplary embodiment, after sending a stop instruction to the target model to stop the target model from generating the output information, the method further includes: if the guiding word is not obtained, sending a regeneration instruction to the target model to regenerate the output information based on the input information, and generating risk warning information corresponding to the regenerated output information.

[0023] According to another embodiment of this application, a device for predicting the category of output information is also provided, comprising: an acquisition module, configured to acquire first content tendency values ​​corresponding to all processing levels within the target model when the target model has completed forward propagation of the input information and has not generated the first output lexical unit corresponding to the input information, wherein the first content tendency value is a scalar value obtained by projection transformation of the hidden state vector of each processing level; a sorting module, configured to arrange multiple first content tendency values ​​based on the processing depth of each processing level to generate a numerical sequence characterizing the change of the first content tendency value with the processing depth; and a prediction module, configured to perform convergence operation on the numerical sequence to obtain a comprehensive representative value, and input the comprehensive representative value into a preset classifier, and predict the category of the output information to be generated by the target model according to the preset classifier, wherein the category of the output information includes: factual output information and non-factual output information.

[0024] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, and the computer program is configured to execute the prediction method for the category of the output information described above when it is run.

[0025] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the above-described method for predicting the category of output information through the computer program.

[0026] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program, wherein the computer program is executed by a processor to perform the above-described method for predicting the category of output information.

[0027] In this embodiment, after the target model completes the forward propagation of the input information but has not generated the first output lexical unit corresponding to the input information, the first content tendency value corresponding to each processing level within the target model is obtained. The first content tendency value is a variable value obtained by projecting the hidden state vector of each processing level. Multiple first content tendency values ​​are arranged based on the processing depth of each processing level to generate a numerical sequence representing the change of the first content tendency value with processing depth. The numerical sequence is aggregated to obtain a comprehensive representative value, which is then input into a preset classifier. The preset classifier predicts whether the output information to be generated by the target model is factual or non-factual. This embodiment solves the problem in related technologies where only the static state of a single processing depth of a large model is read, leading to an inability to accurately determine the authenticity of the output content of the large model, thus enabling accurate determination of the authenticity of the output content of the large model. Attached Figure Description

[0028] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0029] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. It is understood that these drawings only show some embodiments of this application. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0030] Figure 1 This is a hardware structure block diagram of a computer terminal for a method of predicting the category of output information according to an embodiment of this application.

[0031] Figure 2 This is a flowchart of a method for predicting the category of output information according to an embodiment of this application;

[0032] Figure 3 This is a schematic diagram of a hallucination detection method according to an optional embodiment of this application;

[0033] Figure 4 This is a structural block diagram of a device for predicting the category of output information according to an embodiment of this application. Detailed Implementation

[0034] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0035] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0036] The methods and embodiments provided in this application can be executed on a computer terminal or similar computing system. Taking running on a computer terminal as an example, Figure 1 This is a hardware structure block diagram of a computer terminal for a method of predicting the category of output information according to an embodiment of this application. Figure 1 As shown, a computer terminal may include one or more ( Figure 1 (Only one is shown in the image) A processor 102 and a memory 104 for storing data. In an exemplary embodiment, the computer terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the computer terminal described above. For example, the computer terminal may also include components that are more complex than those described above. Figure 1 The more or fewer components shown, or having the same Figure 1 Equivalent functions or ratios shown Figure 1 The functions shown have more different configurations.

[0037] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the method for predicting the category of output information in this embodiment. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thereby implementing the methods described above. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage systems, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0038] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the computer terminal. In one example, the transmission device 106 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet.

[0039] This embodiment provides a method for predicting the category of output information, applied in the aforementioned computer terminal. Figure 2 This is a flowchart of a method for predicting the category of output information according to an embodiment of this application. The process includes the following steps:

[0040] Step S202: When the target model completes the forward propagation of the input information and has not generated the first output word corresponding to the input information, obtain the first content tendency value corresponding to all processing levels inside the target model, wherein the first content tendency value is a scalar value obtained by projection transformation of the hidden state vector of each processing level.

[0041] The target model mentioned above is: a large language model.

[0042] Step S204: Arrange multiple first content tendency values ​​based on the processing depth of each processing level to generate a numerical sequence that characterizes the change of the first content tendency value with processing depth.

[0043] Step S206: Perform convergence operation on the numerical sequence to obtain a comprehensive representative value, and input the comprehensive representative value into a preset classifier. Predict the category of the output information that the target model will generate based on the preset classifier. The categories of the output information include: factual output information and non-factual output information.

[0044] Through the above steps, after the target model completes the forward propagation of the input information and has not generated the first output lexical unit corresponding to the input information, the first content tendency value corresponding to each processing level within the target model is obtained. The first content tendency value is the variable value obtained by projecting the hidden state vector of each processing level. Multiple first content tendency values ​​are arranged based on the processing depth of each processing level to generate a numerical sequence representing the change of the first content tendency value with processing depth. The numerical sequence is aggregated to obtain a comprehensive representative value, which is then input into a preset classifier. The preset classifier predicts whether the output information to be generated by the target model is factual or non-factual. This embodiment solves the problem in related technologies where only the static state of a single processing depth of a large model is read, leading to an inability to accurately determine the authenticity of the output content of a large model, thus enabling accurate determination of the authenticity of the output content of a large model.

[0045] Optionally, obtaining the first content tendency value corresponding to each of the processing layers within the target model includes: obtaining a first hidden state vector at the output of the attention module of each processing layer or at the output of each processing layer, wherein the first hidden state vector is the hidden state vector generated by the target model at each processing layer at the time step corresponding to the last word of the input information; and performing a projection transformation on each first hidden state vector based on a preset projection rule corresponding to each processing layer to obtain the first content tendency value corresponding to each processing layer.

[0046] It is understandable that during the processing of input information from the target object in a large language model, the input information flows sequentially through multiple sequentially arranged Transformer processing levels. This embodiment of the application needs to extract the first content preference value of each processing level, rather than relying solely on one of the processing levels. Specifically:

[0047] During the forward propagation of the large language model, the output of the attention module or the output of each processing layer is located. The first hidden state vector (a high-dimensional vector) obtained at the output of the attention module or the output of each processing layer cannot be directly used to determine truth or falsehood due to its high dimensionality and the presence of irrelevant information (such as tone and sentence structure). Therefore, it is necessary to use a linear projection transformation on the first hidden state vector corresponding to each processing layer using a projection rule obtained through pre-training, so as to map the information in the high-dimensional space to a low-dimensional space and obtain the first content tendency value.

[0048] For example, consider a large language model (LLM) consisting of 12 layers. When the LLM processes the input information: the Eiffel Tower is located in London: at layer 1, it might only recognize the Eiffel Tower as a place name entity; at layer 6, the LLM begins to associate the Eiffel Tower with knowledge of Paris; at layer 11, the LLM detects a mismatch between London and the Eiffel Tower in its internal knowledge base. At this point, it is necessary to extract the current internal state vector (i.e., the first hidden state vector) from each attention head from layer 1 to layer 12, which contains rich semantic, syntactic, and factual cues. Assume the hidden state vector of layer 1 is H1, and the preset projection vector is W1. Calculate the dot product: The result S1 is the first content bias value for this processing level. If S1 is a large positive number, it indicates that the internal state of the first level strongly leans towards factuality; if it is negative, it leans towards non-factuality or uncertainty.

[0049] According to the embodiments of this application, by obtaining the hidden state vector of each processing layer, the complete numerical sequence of the large language model at the shallow (syntax / entity recognition), middle (semantic association), and deep (fact conflict detection) levels is preserved. This avoids the problem of discarding potential discriminative information from other layers due to reading only one layer (such as the last layer).

[0050] Optionally, the first hidden state vector is obtained at the output of the attention module at each processing level or at the output of each processing level, including at least one of the following methods:

[0051] (1) Determine the target attention head for each processing level, wherein the attention module of each processing level includes: the target attention head; obtain the first output vector of the target attention head at the time step corresponding to the last word of the input information, and determine the first output vector as the first hidden state vector of each processing level.

[0052] (2) Obtain the second output vector of at least two attention heads of each processing level at the time step corresponding to the last word of the input information, wherein the attention module of each processing level includes at least two attention heads of each processing level; concatenate or weightedly sum the second output vectors of the at least two attention heads to obtain an aggregate vector, and determine the aggregate vector as the first hidden state vector of each processing level.

[0053] (3) Obtain the third output vector of each processing level after residual connection and normalization operation through the output terminal, and determine the third output vector as the first hidden state vector of each processing level.

[0054] The attention head is responsible for capturing the relationships between different words in the input information, and the output vector of the attention head contains the depth of understanding of the current context by this processing level.

[0055] Understandably, one way to obtain the first hidden state vector is:

[0056] Among the multiple attention heads in each Transformer layer, the target attention head that contributes the most to the factual judgment is determined through pre-training or screening mechanisms; only the first output vector of the target attention head at the time step corresponding to the last word of the input information is read, and then the first output vector is determined as the hidden state vector.

[0057] Method 2 for obtaining the first hidden state vector:

[0058] Since a single attention head may only capture local features, while the combination of multiple attention heads can provide more comprehensive contextual information, any two or more attention heads in each processing layer can be obtained; the second output vectors of each attention head are concatenated or weighted and summed to obtain the first hidden state vector.

[0059] Method 3 for obtaining the first hidden state vector:

[0060] Obtain the third output vector of each processing layer after passing through the attention mechanism, feedforward neural network, residual connection and layer normalization, and determine the third output vector as the first hidden state vector.

[0061] The third output vector is a comprehensive representation of this processing level after a complete nonlinear transformation. It integrates the global dependency of the attention mechanism and the feature transformation of the feedforward network, and also contains the highest level of abstract semantic information.

[0062] Optionally, before projecting each first hidden state vector based on the preset projection rule corresponding to each processing level, the method further includes: constructing a set of sample pairs containing labels, wherein each sample pair in the set contains a sample input information and a sample output information, each sample pair corresponds to a label, the label being used to indicate whether the sample output information in each sample pair is factual output information or non-factual output information; inputting the sample input information in each sample pair into the target model after freezing parameters to determine the second hidden state vector during the processing of the sample input information by each processing level, wherein the second hidden state vector and the first hidden state vector are taken from the same processing position and the same word position, and the second hidden state vector and the first hidden state vector are... The vectors have the same dimension; a projection vector for each processing level is constructed, and a projection transformation is performed on each second hidden state vector based on the projection vector of each processing level to generate a second content tendency value corresponding to each processing level, wherein the dimension of the projection vector is the same as the dimension of the second hidden state vector; based on each loss function, the projection vector of each processing level is updated through a backpropagation algorithm until a preset termination condition is met, and a preset projection rule for each processing level is determined according to each updated projection vector, wherein the preset termination condition includes at least one of the following: the number of updates of the projection vector reaches a preset number, the sum of the loss function values ​​corresponding to all processing levels is less than a preset loss threshold, and in two adjacent iterations of the update of the projection vector, the norm change of the projection vectors corresponding to all processing levels is less than a preset change threshold.

[0063] Specifically, the projection transformation of each second hidden state vector based on the projection vector of each processing level to generate the second content tendency value corresponding to each processing level includes: performing a dot product operation between the projection vector of each processing level and the second hidden state vector of each processing level to obtain the second content tendency value in scalar form.

[0064] The loss function can be of one of the following types: binary cross-entropy loss function or contrastive loss function.

[0065] It is understandable that the technical solution for constructing preset projection rules is as follows:

[0066] A set of sample pairs is collected, comprising multiple sample pairs. Each sample pair includes input information (e.g., a question or context) and output information (e.g., an answer generated by a large language model or a given factual statement). Each sample pair is assigned a label indicating whether it is a factual or non-factual sample pair. For example: Factual sample pair: Input information: Where is the capital of France? Output information: Paris? Label: 1 (indicating a factual sample pair, i.e., the output information in the sample pair is factual). Non-factual sample pair: Input information: Where is the capital of France? Output information: Berlin? Label: 0 (indicating a non-factual sample pair, i.e., the output information in the sample pair is non-factual).

[0067] The sample input information is fed into the large language model, with all parameters of the large language model remaining unchanged. The large language model performs forward propagation. Before generating the final output information after processing the sample input information, the second hidden state vector of each processing level is obtained. This vector indicates the internal cognitive state of the large language model regarding the sample input information during the training phase. A projection vector is initialized for each processing level. This projection vector can be a low-dimensional parameter vector, used to extract the information most relevant to authenticity from the second hidden state vector of the high-dimensional state. The second hidden state vector of each processing level is then multiplied by the corresponding projection vector using a dot product or linear transformation to obtain a scalar value, which is the second content bias value. This value quantifies the degree to which the internal state of this processing level leans towards authenticity.

[0068] Furthermore, the second content bias value is compared with the label of the sample pair. If the label is factual but the second content bias value is negative, a large error occurs. The gradient of the loss function with respect to the projection vector is calculated using the backpropagation algorithm. The loss function can include binary cross-entropy loss or contrastive loss. The parameters of the projection vector are adjusted in the opposite direction of the gradient so that the projection result is closer to the true label in the next calculation. The above steps are repeated until all projection vectors converge, meaning all projection vectors can accurately map the internal state to a bias value reflecting truthfulness. The preset projection rule is then determined based on the finally determined parameters.

[0069] Optionally, the numerical sequence is aggregated to obtain a comprehensive representative value; including at least one of the following methods:

[0070] Method 1: Calculate the sum of multiple first content tendency values ​​contained in the numerical sequence; calculate the arithmetic mean of multiple first content tendency values ​​contained in the numerical sequence based on the sum of the first content tendency values ​​and the number of first content tendency values ​​contained in the numerical sequence, and determine the arithmetic mean as the comprehensive representative value.

[0071] Method 2: Determine the median among the multiple first content tendency values ​​contained in the numerical sequence, and determine the median as the comprehensive representative value.

[0072] Method 3: Determine the weight corresponding to each first content tendency value contained in the numerical sequence, and perform a weighted average of multiple first content tendency values ​​contained in the numerical sequence according to the weight to obtain the comprehensive representative value.

[0073] Method 4: Extract at least one trajectory summarization parameter from the numerical sequence, wherein the trajectory summarization parameter includes at least one of the following: a dispersion parameter characterizing the fluctuation amplitude of the numerical sequence, a trend parameter characterizing the changing trend of the numerical sequence, and an endpoint difference parameter characterizing the difference between the initial state and the final state of the numerical sequence; combine the at least one trajectory summarization parameter into a target feature vector, and map the target feature vector to the comprehensive representative value.

[0074] The comprehensive representative value is then input into a preset classifier, and the category of the output information to be generated by the target model is determined based on the output result of the preset classifier. The preset classifier is a logistic regression classifier.

[0075] Understandably, in order to extract the most stable discrimination criterion from a trajectory containing multiple noises and signals, it is necessary to perform convergence operations on the numerical sequence. These convergence operations can be performed using at least one of the following methods:

[0076] Method 1: Arithmetic mean method.

[0077] Add the first content preference values ​​of all processing levels together, and then calculate the quotient of the sum and the level to obtain the comprehensive representative value.

[0078] Method 2: Median method.

[0079] Sort the multiple primary content preference values ​​from smallest to largest, and take the value in the middle position as the comprehensive representative value.

[0080] Method 3: Weighted average method.

[0081] Assign a weight to the first content tendency value of each layer (e.g., deeper layers have higher weights, or certain specific layers have higher weights), and calculate the weighted sum to obtain the comprehensive representative value.

[0082] Method 4: Calculation based on trajectory generalization parameters:

[0083] Extract at least one trajectory generalization parameter from the numerical sequence, such as: a dispersion parameter representing the fluctuation amplitude of the numerical sequence, a trend parameter representing the changing trend of the numerical sequence, and an endpoint difference parameter representing the difference between the initial and final states of the numerical sequence; combine at least one trajectory generalization parameter into a target feature vector, and map the target feature vector into a comprehensive representative value. The combination method can be a simple concatenation.

[0084] The comprehensive representative value is input into a pre-trained classifier, which generates the category of the output information based on the magnitude of the comprehensive representative value.

[0085] Optionally, after predicting the category of the output information to be generated by the target model according to the preset classifier, the method further includes: if the output information is determined to be non-factual output information, sending a stop instruction to the target model to stop the target model from generating the output information; obtaining the guide word corresponding to the input information, and generating target output information according to the guide word and the input information; inputting the target output information into the target model; sending a generation instruction to the target model to generate target output information according to the guide word and the input information, and determining the category of the target output information before the target model generates the target output information, wherein the category of the target output information includes: factual output information and non-factual output information; if the target output information is determined to be factual output information, instructing the target model to generate the target output information.

[0086] The guiding words are obtained through at least one of the following methods: obtaining the guiding words from the target knowledge base; obtaining the guiding words from a preset fact template; or obtaining the guiding words from a historical factual output information database.

[0087] Understandably, if the information to be output is determined to be non-factual, a stop command is sent to the running large language model. Guiding words are extracted from the target knowledge base, preset fact templates, and historical factual output information. These guiding words are combined with the input information, and the combined input information is fed into the large language model. The large language model then generates output information based on this combined input information. Before generating output information, the first content tendency value of each processing level of the target model is obtained again, and a numerical sequence is generated accordingly. The category of the target output information is determined based on this numerical sequence. If the category is determined to be factual output information, the large language model is instructed to generate this output information.

[0088] Optionally, after sending a stop instruction to the target model to stop the target model from generating the output information, the method further includes: if the guiding word is not obtained, sending a regeneration instruction to the target model to regenerate the output information based on the input information, and generating risk warning information corresponding to the regenerated output information.

[0089] Understandably, in the absence of a guide word, a regeneration instruction is sent to the large language model. This regeneration instruction includes at least one of the following: an instruction to increase randomness, an instruction to change the seed, or an instruction to limit length. The instruction to increase randomness instructs the target model to raise its temperature parameters; the instruction to change the seed instructs the model to reset the random seed, allowing it to try different generation paths; and the instruction to limit length instructs the model to generate shorter responses (less than a preset length), reducing the probability of accumulated illusions in long texts. A risk warning message is generated, which can be placed above or below the regenerated output, indicating the limitations and uncertainties of the regenerated output.

[0090] In summary, the following problems can be solved through the embodiments of this application:

[0091] 1) Related technologies only read a single intermediate layer representation, thus failing to capture the dynamic process of reality being gradually constructed in the Transformer hierarchy;

[0092] 2) The real signals occupy only a tiny variance in the extremely high-dimensional representation space, and the decision boundaries of each layer are nearly orthogonal. The embodiments of this application can solve the problem of how to extract stable and consistent category information from these weak and isolated cross-layer signals.

[0093] 3) The embodiments of this application can solve the problem of single-layer reading being susceptible to random noise and local disturbance interference;

[0094] 4) The embodiments of this application can solve the problem that the highly redundant reading of the same signal by multiple attention heads within a layer cannot truly improve the signal-to-noise ratio;

[0095] 5) In the absence of theoretical guidance for the effective combination of massive cross-layer features, the optional embodiments of this application can solve the problem of how to find a low-dimensional and sufficient detection statistic (e.g., a scalar) to minimize information loss with extremely low computational complexity and prevent overfitting.

[0096] 6) The embodiments of this application can solve the problem that most related methods perform post-detection after generating the sequence. In the embodiments of this application, the detection is completed before the large language model generates any answer text, thereby achieving a qualitative change from post-detection to pre-detection.

[0097] To better understand the process of the above-mentioned method for predicting the category of output information, the flow of the above-mentioned method for predicting the category of output information will be described below in conjunction with optional embodiments, but this is not intended to limit the technical solution of the embodiments of this application.

[0098] An optional embodiment of this application provides a hallucination detection method that, instead of reading signals from a single cross-section of the model, treats the continuous change process of the internal state along the processing depth as a whole, and converges this dynamic trajectory to obtain a more stable and sufficient discrimination criterion. Specifically:

[0099] Before the large language model (i.e., the target model) understands the input question (i.e., input information) and prepares to generate an answer (i.e., output information), a set of feature representations (i.e., first content bias values) reflecting the current internal state are extracted from multiple processing layers of different depths within the model. The internal locations or components from which these features are extracted can be fixed, such as the output of a specific functional unit. When reading the internal state from each layer, the object read is not limited to a single functional unit; it can also be a synthesis of information from multiple units of the same type within the same layer, or it can directly read the global state of that layer. These different sources of information essentially carry similar, highly shared discriminative information features.

[0100] The specific extraction rules (i.e., preset projection rules) are accomplished through pre-trained mappings, which can compress the high-dimensional representation of this level into a numerical value that reflects the tendency towards realism. The preset projection rules used to map high-dimensional states to a single-tendency numerical value can also have different learning forms or regularization methods, as long as they can extract the direction related to realism while maintaining generalization ability.

[0101] Along the depth direction of processing, a series of values ​​are obtained, forming a numerical sequence reflecting changes in internal cognition. It is important to note that not every detail change in this numerical sequence contributes equally to the final judgment; instead, most of the energy from random fluctuations at different depths is task-independent perturbation. Based on a deeper analysis of this characteristic, a comprehensive, uniform merging of these values ​​(i.e., convergence operation) can be performed to obtain a single, comprehensive scalar index. In the information convergence stage, in addition to uniform averaging across all depths, some equally simple and physically meaningful overall trajectory generalization parameters can be used, such as overall fluctuation amplitude, differences between start and end points, or main trends. Combining one or several of these values ​​into a very low-dimensional feature vector can achieve detection power nearly identical to a single average. Furthermore, the method for generating the numerical sequence can be smoothly extended from a single snapshot before generation to the generation process, continuously monitoring changes in internal tendency indicators over time to form a sustained risk assessment of the output process.

[0102] Then, using a mapping trained with a small number of parameters, a final judgment can be made about whether the model is about to produce a factual error based on this scalar metric. The entire process is completed before the model outputs any response, allowing the system to intervene in advance and prevent the generation and spread of erroneous content.

[0103] In summary, the above-described technical solutions of the optional embodiments of this application can overcome the limitations of single-point static reading and instead utilize distributed information aggregation in the depth or time dimension to enhance the robustness of the judgment.

[0104] In one exemplary embodiment, Figure 3 This is a schematic diagram of a hallucination detection method according to an optional embodiment of this application, such as... Figure 3 As shown:

[0105] in, Figure 3 The logical value trajectory in the embodiment of this application is the numerical sequence. Figure 3 The probe obtained from the learning process is the preset projection rule of this application embodiment; Figure 3 In this context, "A0-A1" refers to the attention head in the embodiments of this application.

[0106] Before an autoregressive model (i.e., the target model) consisting of multiple sequential processing layers begins generating output content (i.e., output information) based on the received input information, the following steps are performed:

[0107] Step 1, determine a reading location (i.e. Figure 3 Answer starting position This reading position corresponds to the end of the input content.

[0108] By fixing the reading position at the end of the input content, the entire process of state acquisition, sequence generation, convergence calculation, and judgment is completed before the model actually outputs any response text. This transforms the existing passive, post-event detection into proactive prediction, enabling early intervention against factual errors.

[0109] Step two: During the layer-by-layer internal processing of the model, a first content bias value is obtained from the attention head (A0-A1) of each hidden layer. The first content bias value is obtained by using a projection rule pre-trained on samples for each layer to project the high-dimensional state data of that layer into a numerical value. This numerical value is used to characterize the internal bias of that layer towards the content to be generated (i.e., the learned probe).

[0110] Among them, t The hidden vector obtained at the last time step before generation begins, or at a specific moment before decoding, is the first hidden state vector.

[0111] Step 3: Arrange the first content tendency values ​​obtained from each layer according to the processing depth of the layer to form a data sequence (i.e., a numerical sequence). Figure 3 (The logical value trajectory in the text).

[0112] Step four involves performing aggregation operations on the data sequence, such as calculating the average, weighted average, and median, to obtain a comprehensive representative value. Specifically, the aggregation operation calculates the arithmetic mean of all first content tendency values ​​in the data sequence.

[0113] It is understood that the optional embodiments of this application utilize the approximately orthogonal nature of the projection directions between adjacent processing depths. This property makes the random fluctuation components contained in the first content tendency values ​​generated by each layer less correlated in the depth direction. Based on this, by performing a uniform convergence operation on the first content tendency values ​​of each layer, the common stable signal components can be amplified, while the noise components of independent fluctuations in each layer can be smoothed. Furthermore, the optional embodiments of this application establish a scheme using the sequence arithmetic mean as the detection statistic, and theoretically establish the near-optimality of this mean among all linear convergence methods. Only a single scalar value can characterize the discriminative information of the entire sequence, greatly simplifying the detection pipeline and avoiding the overfitting risk and computational overhead caused by complex statistical modeling.

[0114] Step five involves using this comprehensive representative value, and then employing a pre-trained classifier to determine whether the model will produce non-factual output. This determination is made before the model generates any output.

[0115] In an optional embodiment of this application, the specific location within the model used to obtain the state value can be the output of a functional unit in each processing layer, or the total output of the entire processing layer. The projection rule used to obtain the first content bias value has its internal parameters determined during the training phase, and the goal of the training is to enable the first content bias value to distinguish between factual and non-factual internal states.

[0116] An optional embodiment of this application also provides an illusion detection device, comprising a memory storing pre-trained projection rule parameters and classifier parameters, and a processor. The processor is configured to acquire the internal states of each layer of the model at the read position, calculate a first content tendency value for each layer using the stored parameters, average the first content tendency values, and then input the average value into the classifier to obtain a judgment signal. This device is connected to the model to be detected via an interface without modifying the structure of the model itself.

[0117] In one specific embodiment, using a certain open-source large language model (Qwen2.5-72B) as the target model, the method of this application embodiment is compared with probe methods based on a single processing layer in related technologies on an intelligent agent task dataset. The test results show that the best-performing single-layer probe method in related technologies has an Area Under the Receiver Operating Characteristic Curve (AUROC) of 86.16% and an Area Under the Precision-Recall Curve (AUPRC) of 85.16%; while the method of this application embodiment achieves an AUROC of 97.86% and an AUPRC of 94.94%, representing an improvement of 11.70 percentage points in AUROC and 9.78 percentage points in AUPRC compared to the aforementioned single-layer probe method.

[0118] In summary, the optional embodiments of this application obtain state values ​​from multiple processing layers and form a data sequence along the depth direction, thus converging and utilizing the scattered information from each layer. Compared with the single-layer detection schemes in related technologies, the detection accuracy of the optional embodiments of this application is significantly improved in various models and tasks, especially in complex task scenarios requiring multi-step reasoning, where the error rate is reduced by more than 80%. Furthermore, the optional embodiments of this application establish the near-optimality of the sequence mean, providing a detection statistic selection standard with a clear theoretical basis. This allows the optional embodiments of this application to break free from the dependence on empirical parameter tuning, and complete the judgment using only a single scalar value, avoiding the overfitting risk caused by using complex high-dimensional features for classification. Moreover, the embodiments of this application fix the reading position at the end of the input content, and the entire detection process is completed before the first character is generated, giving the system the opportunity to take measures before erroneous content is generated, preventing it from continuing to be generated or output to the user, thus achieving preventive intervention. The optional embodiments of this application establish a complete logical chain from phenomenon to solution by theoretically analyzing and empirically verifying the sparsity of signal distribution, the quasi-orthogonality of projection directions of adjacent layers, and the dominance of sequence mean in the sense of signal-to-noise ratio.

[0119] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0120] Figure 4 This is a structural block diagram of a device for predicting the category of output information according to an embodiment of this application; as shown below. Figure 4 As shown, it includes:

[0121] Extraction module 42 is used to obtain the first content tendency value corresponding to each of the processing levels inside the target model when the target model has completed the forward propagation of the input information and has not generated the first output word corresponding to the input information. The first content tendency value is a scalar value obtained by projection transformation of the hidden state vector of each processing level.

[0122] The sorting module 44 is used to sort multiple first content tendency values ​​based on the processing depth of each processing level to generate a numerical sequence that characterizes the change of the first content tendency value with the processing depth.

[0123] The determination module 46 is used to perform convergence operation on the numerical sequence to obtain a comprehensive representative value, and input the comprehensive representative value into a preset classifier. Based on the preset classifier, the module predicts the category of the output information that the target model will generate. The categories of the output information include: factual output information and non-factual output information.

[0124] According to the above apparatus, when the target model completes the forward propagation of the input information but has not generated the first output lexical unit corresponding to the input information, the first content tendency value corresponding to each processing level within the target model is obtained. The first content tendency value is a variable value obtained by projecting the hidden state vector of each processing level. Multiple first content tendency values ​​are arranged based on the processing depth of each processing level to generate a numerical sequence representing the change of the first content tendency value with processing depth. The numerical sequence is aggregated to obtain a comprehensive representative value, which is then input into a preset classifier. The preset classifier predicts whether the output information to be generated by the target model is factual or non-factual. Through this embodiment, the problem in related technologies where only the static state of a single processing depth of a large model is read, leading to an inability to accurately determine the authenticity of the output content of a large model, is solved, thereby accurately determining the authenticity of the output content of a large model.

[0125] In an exemplary embodiment, the acquisition module 42 is further configured to acquire a first hidden state vector at the output of the attention module of each processing level or at the output of each processing level, wherein the first hidden state vector is a hidden state vector generated by the target model at each processing level at the time step corresponding to the last word of the input information; and to perform a projection transformation on each first hidden state vector based on a preset projection rule corresponding to each processing level to obtain a first content tendency value corresponding to each processing level.

[0126] In an exemplary embodiment, the acquisition module 42 is further configured to determine the target attention head for each processing level, wherein the attention module for each processing level includes: the target attention head; acquiring the first output vector of the target attention head at the time step corresponding to the last lexical term of the input information, and determining the first output vector as the first hidden state vector for each processing level.

[0127] In an exemplary embodiment, the acquisition module 42 is further configured to acquire the second output vector of at least two attention heads of each processing level at the time step corresponding to the last word of the input information, wherein the attention module of each processing level includes at least two attention heads of each processing level; the second output vectors of the at least two attention heads are concatenated or weighted to obtain an aggregate vector, and the aggregate vector is determined as the first hidden state vector of each processing level.

[0128] In an exemplary embodiment, the acquisition module 42 is further configured to acquire the third output vector output by the output terminal of each processing level after residual connection and normalization operations, and determine the third output vector as the first hidden state vector of each processing level.

[0129] In an exemplary embodiment, the acquisition module 42 is further configured to construct a sample pair set containing labels, wherein each sample pair in the sample pair set contains a sample input information and a sample output information, each sample pair corresponds to a label, and the label is used to indicate whether the sample output information in each sample pair is factual output information or non-factual output information; input the sample input information in each sample pair into the target model after freezing parameters to determine the second hidden state vector in the process of each processing level processing the sample input information, wherein the second hidden state vector and the first hidden state vector are taken from the same processing position and the same word position, and the second hidden state vector and the first hidden state vector have the same dimension; construct the projection vector of each processing level, and based on the projection vector of each processing level... The quantity is projected onto each second hidden state vector to generate a second content bias value corresponding to each processing level, wherein the dimension of the projection vector is the same as the dimension of the second hidden state vector; a loss function is constructed for each projection vector based on the second content bias value corresponding to each processing level and the label of each sample pair; based on each loss function, the projection vector of each processing level is updated through backpropagation algorithm until a preset termination condition is met, and a preset projection rule for each processing level is determined according to each updated projection vector, wherein the preset termination condition includes at least one of the following: the number of updates of the projection vector reaches a preset number, the sum of the loss function values ​​corresponding to all processing levels is less than a preset loss threshold, and in two adjacent iterations of the update of the projection vector, the norm change of the projection vectors corresponding to all processing levels is less than a preset change threshold.

[0130] In an exemplary embodiment, the acquisition module 42 is further configured to perform a dot product operation between the projection vector of each processing level and the second hidden state vector of each processing level to obtain a second content tendency value in scalar form.

[0131] In one exemplary embodiment, the loss function may be of one of the following types: binary cross-entropy loss function or contrastive loss function.

[0132] In one exemplary embodiment, the preset classifier is a logistic regression classifier.

[0133] In an exemplary embodiment, the prediction module 46 is further configured to calculate the sum of a plurality of first content tendency values ​​contained in the numerical sequence; calculate the arithmetic mean of the plurality of first content tendency values ​​contained in the numerical sequence based on the sum of the plurality of first content tendency values ​​and the number of first content tendency values ​​contained in the numerical sequence, and determine the arithmetic mean as the comprehensive representative value.

[0134] In an exemplary embodiment, the prediction module 46 is further configured to determine the median of a plurality of first content tendency values ​​contained in the numerical sequence, and to determine the median as the comprehensive representative value.

[0135] In an exemplary embodiment, the prediction module 46 is further configured to determine the weight corresponding to each first content tendency value contained in the numerical sequence; and to perform a weighted average of the multiple first content tendency values ​​contained in the numerical sequence according to the weights to obtain the comprehensive representative value.

[0136] In an exemplary embodiment, the prediction module 46 is further configured to extract at least one trajectory generalization parameter from the numerical sequence, wherein the trajectory generalization parameter includes at least one of the following: a dispersion parameter characterizing the fluctuation amplitude of the numerical sequence, a trend parameter characterizing the changing trend of the numerical sequence, and an endpoint difference parameter characterizing the difference between the starting state and the ending state of the numerical sequence; combining the at least one trajectory generalization parameter into a target feature vector, and mapping the target feature vector to the comprehensive representative value.

[0137] In an exemplary embodiment, the prediction module 46 is further configured to: send a stop instruction to the target model to stop generating the output information when it is determined that the output information is non-factual output information; obtain the guiding word corresponding to the input information and generate target input information based on the guiding word and the input information; input the target input information into the target model; send a generation instruction to the target model to generate target output information based on the guiding word and the input information, and determine the category of the target output information before the target model generates the target output information, wherein the category of the target output information includes: factual output information and non-factual output information; and instruct the target model to generate the target output information when it is determined that the target output information is factual output information.

[0138] In an exemplary embodiment, the prediction module 46 is further configured to obtain the guiding word by at least one of the following methods: obtaining the guiding word from a target knowledge base; obtaining the guiding word from a preset fact template; or obtaining the guiding word from a historical factual output information database.

[0139] In an exemplary embodiment, the prediction module 46 is further configured to send a regeneration instruction to the target model when the guiding word is not obtained, so that the target model regenerates the output information based on the input information and generates risk warning information corresponding to the regenerated output information.

[0140] Embodiments of this application also provide a storage medium including a stored program, wherein the program executes any of the methods described above when it is run.

[0141] Optionally, in this embodiment, the storage medium may be configured to store program code for performing the following steps:

[0142] S1, when the target model has completed the forward propagation of the input information and has not generated the first output word corresponding to the input information, obtain the first content tendency value corresponding to all processing levels inside the target model, wherein the first content tendency value is a scalar value obtained by projection transformation of the hidden state vector of each processing level.

[0143] S2, Arrange multiple first content tendency values ​​based on the processing depth of each processing level to generate a numerical sequence that characterizes the change of the first content tendency value with processing depth;

[0144] S3, perform aggregation operation on the numerical sequence to obtain a comprehensive representative value, and input the comprehensive representative value into a preset classifier. Based on the preset classifier, predict the category of the output information that the target model will generate. The categories of the output information include: factual output information and non-factual output information.

[0145] Embodiments of this application also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0146] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0147] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0148] S1, when the target model has completed the forward propagation of the input information and has not generated the first output word corresponding to the input information, obtain the first content tendency value corresponding to all processing levels inside the target model, wherein the first content tendency value is a scalar value obtained by projection transformation of the hidden state vector of each processing level.

[0149] S2, Arrange multiple first content tendency values ​​based on the processing depth of each processing level to generate a numerical sequence that characterizes the change of the first content tendency value with processing depth;

[0150] S3, perform aggregation operation on the numerical sequence to obtain a comprehensive representative value, and input the comprehensive representative value into a preset classifier. Based on the preset classifier, predict the category of the output information that the target model will generate. The categories of the output information include: factual output information and non-factual output information.

[0151] Embodiments of this application also provide a computer program product, including a computer program that is executed by a processor through the steps of any of the above method embodiments.

[0152] Optionally, in this embodiment, the above-mentioned computer program product can be executed by a processor using the following steps:

[0153] S1, when the target model has completed the forward propagation of the input information and has not generated the first output word corresponding to the input information, obtain the first content tendency value corresponding to all processing levels inside the target model, wherein the first content tendency value is a scalar value obtained by projection transformation of the hidden state vector of each processing level.

[0154] S2, Arrange multiple first content tendency values ​​based on the processing depth of each processing level to generate a numerical sequence that characterizes the change of the first content tendency value with processing depth;

[0155] S3, perform aggregation operation on the numerical sequence to obtain a comprehensive representative value, and input the comprehensive representative value into a preset classifier. Based on the preset classifier, predict the category of the output information that the target model will generate. The categories of the output information include: factual output information and non-factual output information.

[0156] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0157] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0158] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0159] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for predicting the category of output information, characterized in that, include: When the target model completes the forward propagation of the input information and has not generated the first output word corresponding to the input information, the first content tendency value corresponding to each of the processing layers inside the target model is obtained, wherein the first content tendency value is a scalar value obtained by projection transformation of the hidden state vector of each processing layer. Multiple first content tendency values ​​are arranged based on the processing depth of each processing level to generate a numerical sequence characterizing the change of the first content tendency values ​​with processing depth. The numerical sequence is aggregated to obtain a comprehensive representative value, which is then input into a preset classifier. The preset classifier is used to predict the category of the output information that the target model will generate. The categories of the output information include: factual output information and non-factual output information.

2. The method for predicting the category of output information according to claim 1, characterized in that, Obtain the first content preference value corresponding to each of the processing levels within the target model, including: A first hidden state vector is obtained at the output of the attention module of each processing layer or at the output of each processing layer, wherein the first hidden state vector is the hidden state vector generated by the target model at each processing layer at the time step corresponding to the last word of the input information. Based on the preset projection rules corresponding to each processing level, a projection transformation is performed on each first hidden state vector to obtain the first content tendency value corresponding to each processing level.

3. The method for predicting the category of output information according to claim 2, characterized in that, Obtaining the first hidden state vector at the output of the attention module at each processing level or at the output of each processing level includes: Determine the target attention head for each processing level, wherein the attention module of each processing level includes the target attention head; Obtain the first output vector of the target attention head at the time step corresponding to the last word of the input information, and determine the first output vector as the first hidden state vector of each processing layer.

4. The method for predicting the category of output information according to claim 2, characterized in that, Obtaining the first hidden state vector at the output of the attention module of each processing layer or at the output of each processing layer further includes: Obtain the second output vector of at least two attention heads of each processing level at the time step corresponding to the last word of the input information, wherein the attention module of each processing level includes at least two attention heads of each processing level; The second output vectors of the at least two attention heads are concatenated or weighted and summed to obtain an aggregate vector, and the aggregate vector is determined as the first hidden state vector of each processing layer.

5. The method for predicting the category of output information according to claim 2, characterized in that, Obtaining the first hidden state vector at the output of the attention module of each processing layer or at the output of each processing layer further includes: Obtain the third output vector of each processing level after residual connection and normalization operations, and determine the third output vector as the first hidden state vector of each processing level.

6. The method for predicting the category of output information according to claim 2, characterized in that, Before performing projection transformation on each first hidden state vector based on the preset projection rules corresponding to each processing level, the method further includes: Construct a set of sample pairs containing labels, wherein each sample pair in the set contains a sample input information and a sample output information, and each sample pair corresponds to a label, the label being used to indicate whether the sample output information in each sample pair is factual output information or non-factual output information; The sample input information in each sample pair is input into the target model after freezing parameters to determine the second hidden state vector in the process of each processing level processing the sample input information. The second hidden state vector and the first hidden state vector are taken from the same processing position and the same word position, and the second hidden state vector and the first hidden state vector have the same dimension. Construct a projection vector for each processing level, and perform a projection transformation on each second hidden state vector based on the projection vector of each processing level to generate a second content tendency value corresponding to each processing level, wherein the dimension of the projection vector is the same as the dimension of the second hidden state vector; A loss function is constructed for each projection vector based on the second content bias value corresponding to each processing level and the label of each sample pair; Based on each loss function, the projection vector of each processing level is updated through backpropagation until a preset termination condition is met. A preset projection rule for each processing level is determined based on each updated projection vector. The preset termination condition includes at least one of the following: the number of updates to the projection vector reaches a preset number; the sum of the loss function values ​​corresponding to all processing levels is less than a preset loss threshold; and in two adjacent iterations of updating the projection vector, the norm change of the projection vectors corresponding to all processing levels is less than a preset change threshold.

7. The method for predicting the category of output information according to claim 6, characterized in that, Based on the projection vector of each processing level, a projection transformation is performed on each second hidden state vector to generate a second content tendency value corresponding to each processing level, including: Perform a dot product operation between the projection vector of each processing level and the second hidden state vector of each processing level to obtain a second content tendency value in scalar form.

8. The method for predicting the category of output information according to claim 6, characterized in that, The loss function can be of one of the following types: binary cross-entropy loss function or contrastive loss function.

9. The method for predicting the category of output information according to claim 1, characterized in that, The preset classifier is a logistic regression classifier.

10. The method for predicting the category of output information according to claim 1, characterized in that, The numerical sequence is aggregated to obtain a comprehensive representative value, including: Calculate the sum of multiple first content tendency values ​​contained in the numerical sequence; The arithmetic mean of the multiple first content tendency values ​​contained in the numerical sequence is calculated based on the sum of the multiple first content tendency values ​​and the number of first content tendency values ​​contained in the numerical sequence, and the arithmetic mean is determined as the comprehensive representative value.

11. The method for predicting the category of output information according to claim 1, characterized in that, The process of performing a convergence operation on the numerical sequence to obtain a comprehensive representative value also includes: The median of the multiple first content tendency values ​​contained in the numerical sequence is determined, and the median is determined as the comprehensive representative value.

12. The method for predicting the category of output information according to claim 1, characterized in that, The process of performing a convergence operation on the numerical sequence to obtain a comprehensive representative value also includes: Determine the weight corresponding to each first content tendency value contained in the numerical sequence; The comprehensive representative value is obtained by weighting multiple first content tendency values ​​contained in the numerical sequence according to the weights.

13. The method for predicting the category of output information according to claim 1, characterized in that, The process of performing a convergence operation on the numerical sequence to obtain a comprehensive representative value also includes: Extract at least one trajectory summarization parameter from the numerical sequence, wherein the trajectory summarization parameter includes at least one of the following: a dispersion parameter characterizing the fluctuation amplitude of the numerical sequence, a trend parameter characterizing the changing trend of the numerical sequence, and an endpoint difference parameter characterizing the difference between the initial state and the final state of the numerical sequence. The at least one trajectory summarization parameter is combined into a target feature vector, and the target feature vector is mapped to the comprehensive representative value.

14. The method for predicting the category of output information according to claim 1, characterized in that, After predicting the category of the output information to be generated by the target model based on the preset classifier, the method further includes: If it is determined that the output information is non-factual, a stop command is sent to the target model to cause the target model to stop generating the output information; Obtain the guiding words corresponding to the input information, and generate target input information based on the guiding words and the input information; The target input information is input into the target model; A generation instruction is sent to the target model to enable the target model to generate target output information based on the guiding words and the input information. Before the target model generates the target output information, the category of the target output information is determined, wherein the category of the target output information includes: factual output information and non-factual output information. If the target output information is determined to be factual output information, the target model is instructed to generate the target output information.

15. The method for predicting the category of output information according to claim 14, characterized in that, The guide word can be obtained through at least one of the following methods: Obtain the guiding words from the target knowledge base; The guiding words are obtained from the preset fact template; The guiding words are retrieved from the historical factual output information database.

16. The method for predicting the category of output information according to claim 14, characterized in that, After sending a stop command to the target model to stop the target model from generating the output information, the method further includes: If the guiding words are not obtained, a regeneration instruction is sent to the target model so that the target model regenerates the output information based on the input information and generates risk warning information corresponding to the regenerated output information.

17. A device for predicting the category of output information, characterized in that, include: The acquisition module is used to acquire the first content tendency value corresponding to each of the processing layers inside the target model when the target model has completed the forward propagation of the input information and has not generated the first output word corresponding to the input information. The first content tendency value is a scalar value obtained by projection transformation of the hidden state vector of each processing layer. The sorting module is used to arrange multiple first content tendency values ​​based on the processing depth of each processing level to generate a numerical sequence that characterizes the change of the first content tendency value with the processing depth. The prediction module is used to perform convergence operations on the numerical sequence to obtain a comprehensive representative value, and input the comprehensive representative value into a preset classifier. Based on the preset classifier, the module predicts the category of the output information that the target model will generate. The categories of the output information include: factual output information and non-factual output information.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method of any one of claims 1-17.

19. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method of any one of claims 1-17.

20. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method of any one of claims 1 to 17.

Images