Large language model distribution external sample detection method and system suitable for general task
By operating in the semantic event space, estimating the semantic equivalence relationship and calculating the external confidence of the distribution, the problem of inaccurate response of large language models when facing external distribution input is solved, the reliability and accuracy of the model is improved, and false positives and missed reports are reduced.
Patent Information
- Application Number
- CN202510133212.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-06
- Publication Date
- 2025-06-24
AI Technical Summary
Existing large language models cannot respond accurately when facing inputs that are significantly different from the distribution of training data, especially in key areas such as autonomous driving and medical diagnosis.
By operating in the semantic event space, a novel algorithm is introduced to estimate the semantic equivalence relationship and calculate the distribution external confidence, combined with conservative prediction to reduce the false positive rate, and determine whether the input is an out-of-distribution sample.
It significantly improves the response reliability and accuracy of large language models when facing unknown inputs, reduces false positives and missed reports, improves users' perception of model performance, and provides a smoother and more accurate interactive experience.
Smart Images
Figure CN120196712A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of natural language processing, and in particular, to an out-of-distribution sample detection method and system for large language models applicable to general tasks. Background Art
[0002] With the rapid development of artificial intelligence technology, especially in the field of natural language processing, large language models have become key tools for performing complex language tasks. These models, through training on vast datasets, can generate coherent and semantically rich text and are widely applied in various scenarios such as machine translation, text summarization, and question answering systems. However, existing large language models often fail to respond accurately when faced with inputs that are significantly different from the training data distribution (i.e., out-of-distribution samples), which may lead to serious consequences in critical fields such as autonomous driving and medical diagnosis. Although there has been research dedicated to out-of-distribution anomaly detection, it has mainly focused on classification tasks, with relatively less research on generation tasks, and existing methods have significant deficiencies in handling the high-dimensional and multi-class form outputs of large language models. Traditional out-of-distribution detection methods, such as density estimation, threshold methods, isolation forests, etc., usually require strong assumptions about the data distribution and often lack flexibility and accuracy when dealing with high-dimensional data and complex data structures. In addition, the text length generated by large language models is variable, which conflicts with the fixed output dimensions required by many detection methods, making it difficult to directly apply traditional techniques to large language models. In view of this, the present invention aims to propose a general out-of-distribution detection method that is applicable not only to the classification tasks of large language models but also to generation tasks. The present invention operates in a semantic event space, introduces a novel algorithm to estimate semantic equivalence relationships, and calculates out-of-distribution confidence, thereby overcoming the limitations of the prior art. Our detection method incorporates conservative predictions to establish verifiable bounds on the false positive rate. Summary of the Invention
[0003] The present invention proposes an out-of-distribution sample detection method and system for large language models applicable to general tasks, which can significantly improve the performance of the algorithm on various datasets and model scales, especially the accuracy in dealing with far-out-of-distribution and near-out-of-distribution scenarios.
[0004] To achieve the above object, the technical solution of the present invention includes the following content.
[0005] An out-of-distribution sample detection method for large language models applicable to general tasks, the method comprising:
[0006] Based on a large language model, generating an original answer set A for the question, the original answer set A containing multiple answers a to the question;
[0007] Calculating the Shapley value of the original answer set A;
[0008] Based on the Shapley value of the original answer set A, determine whether the problem is an out-of-distribution sample.
[0009] Furthermore, calculating the Shapley value of the original answer set A includes:
[0010] Select an answer subset X from the original answer set A that does not contain the answer a, and add the answer a to the answer subset X to obtain the answer subset X ′ ;
[0011] Calculate the differential entropy of the answer subset X and the answer subset X ′ respectively;
[0012] Based on the change in differential entropy between the answer subset X and the answer subset X ′ obtain the Shapley value of the answer a.
[0013] Based on the Shapley values of all answers in the original answer set A, obtain the Shapley value of the original answer set A.
[0014] Furthermore, calculating the differential entropy of the answer subset X includes:
[0015] Based on the semantic distance between each pair of answers in the original answer set A, construct a semantic equivalence relation matrix W;
[0016] Extract the rows and columns related to the answer subset X from the semantic equivalence relation matrix W to form a submatrix W x ;
[0017] Based on the submatrix W x calculate the differential entropy of the answer subset X.
[0018] Furthermore, the Shapley value where h(v(X)) represents the differential entropy of the answer subset X, v(·) is a function that maps the answer subset X to an n-dimensional random vector, and h(v(X∪{a})) represents the differential entropy of the answer subset X ′ respectively.
[0019] Furthermore, based on the Shapley value of the original answer set A, determining whether the problem is an out-of-distribution sample includes:
[0020] Obtain a set threshold and compare it with the Shapley value of the original answer set A;
[0021] If the Shapley value of the original answer set A is less than the set threshold, determine that the problem is an in-distribution sample;
[0022] If the Shapley value of the original answer set A is greater than the set threshold, it is determined that the problem is an out-of-distribution sample.
[0023] An out-of-distribution sample detection system for large language models applicable to general tasks, the system comprising:
[0024] A generation module, configured to generate an original answer set A for a problem based on a large language model, the original answer set A including multiple answers a to the problem;
[0025] A calculation module, configured to calculate the Shapley value of the original answer set A;
[0026] A detection module, configured to determine whether the problem is an out-of-distribution sample based on the Shapley value of the original answer set A.
[0027] An electronic device, the electronic device comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the out-of-distribution sample detection method for large language models applicable to general tasks according to any one of the above is implemented.
[0028] A computer-readable storage medium, characterized in that computer program instructions are stored on the computer-readable storage medium, and when the computer program instructions are executed by a processor, the out-of-distribution sample detection method for large language models applicable to general tasks according to any one of the above is implemented.
[0029] A computer program product, characterized in that when the computer program product runs on a computer device, the computer device is caused to execute the out-of-distribution sample detection method for large language models applicable to general tasks according to any one of the above.
[0030] Compared with the prior art, the present invention has at least the following beneficial effects.
[0031] By effectively identifying out-of-distribution samples, the present invention reduces the risk of the large language model generating unreliable or potentially dangerous outputs when encountering inputs that do not conform to the training data distribution. This method improves the response reliability of the large language model when facing unknown inputs by evaluating the semantic consistency of the generated text, ensuring the accuracy and stability of the model output. By reducing false positives and false negatives, the present invention improves the user's perception of the performance of the large language model, providing a smoother and more accurate interaction experience. By analyzing the semantic consistency of multiple generated responses, the present invention provides a new framework for out-of-distribution detection, applicable to classification tasks and generation tasks of language models. A detection method with provable guarantees is introduced, which can provide a bound on the false positive rate through order-preserving prediction. By calibrating the multi-inference scores for the calibration set, the present invention can precisely control the false positive rate, ensuring a predefined reliability level when identifying out-of-distribution samples in different language tasks. Description of the Drawings
[0032] Figure 1 Flowchart of the out-of-distribution sample detection method for large language models applicable to general tasks Detailed Implementation Manner
[0033] To understand the purpose, technical solution, and advantages of the present application more clearly, the present invention will be described and illustrated below with reference to the accompanying drawings
[0034] The present invention is constructed based on in-distribution equivalence and semantic equivalence. For example, for any in-distribution sample x ∈ D, where D is the input sample space, that is, the set of all possible input samples. If there is a clear correlation between multiple inference predictions of it and the corresponding transformation g' of its output, then the language model f is considered to have in-distribution equivalence
[0035] Let f represent the function of a large language model (LLM), and x represent a fixed in-distribution input. For multiple inference runs a i ∶=f i (x), where i represents different random instances. If the model f is considered equivalent, the following conditions need to be met: for all i, j
[0036]
[0037] For all i and j, the two inference results f i (x) and f j (x) are semantically equivalent through a certain transformation g ′ This means that even if the two inference results may be different lexically, their semantics are consistent. In other words, this equation emphasizes that even under the influence of randomness, multiple inference results still have the same meaning in the semantic space. For example, the user's question is: "What's the weather like tomorrow?" The system gives the inference result f1(x): "Tomorrow's weather forecast shows it will be sunny." For example, "shows" and "predicts", "will be sunny" and "will be a sunny day", but semantically, they express the same meaning: The weather tomorrow will be sunny. This means that through a certain transformation (such as semantic analysis or semantic similarity model), it can be determined that they are equivalent in the semantic space. Where g' represents the corresponding transformation in the semantic space. This equivalence can be quantified by the semantic equivalence relation w(·|·):
[0038]
[0039] The present invention considers that the answers a i and a j have the same semantic meaning if w(a i ,a j|x) is close to 1, which indicates a high degree of mutual implication between the two, and thus semantic equivalence. In the terms of the semantic equivalence relation w(·,·), when w(a i ,a j |x) = 1 holds for all i, j, invariance occurs. This means that all the generated answers are not only semantically equivalent but also exactly the same lexically, that is, they express the same meaning.
[0040] Figure 1 Shows the detailed steps of the out-of-distribution detection method for large language models.
[0041] Step 1: Based on the large language model, generate the original answer set A for the question, and the original answer set A contains multiple answers a to the question.
[0042] The user poses a question to the language model. This is the starting point of the out-of-distribution detection process. That is, input a question that the user needs to know into the large language model. The language model makes multiple inferences (for example, n = 5 times) for the input query, aiming to generate a series of possible answers. These inferences reflect the model's various possible understandings and responses to the question. After the multi-inference process, the model generates multiple answers, including the correct answer and some possibly incorrect or irrelevant answers.
[0043] Step 2: Calculate the Shapley value of the original answer set A.
[0044] The present invention uses the Shapley value to quantify the contribution of the answer to the overall uncertainty. The Shapley value comes from game theory and is used to fairly distribute the total payoff in a cooperative game. Here, it is used to evaluate the impact of each answer on the overall confidence.
[0045] Step 2.1: Select an answer subset X from the original answer set A that does not contain the answer a, and add the answer a to the answer subset X to obtain the answer subset X ′ .
[0046] The present invention obtains the Shapley value of the original answer set A based on the Shapley value of each answer. And to obtain the Shapley value of each answer, the present invention generates an answer subset X that does not contain the answer a and an answer subset X that contains the answer a respectively ′ .
[0047] Step 2.2: Calculate the differential entropy of the answer subset X and the answer subset X ′ respectively, and based on the differential entropy change of the answer subset X and the answer subset X ′ , obtain the Shapley value of the answer a
[0048] To calculate the differential entropy of the answer subset, for the generated answer set A, the present invention first calculates the semantic distance between each pair of answers to construct a semantic correlation matrix W: Using the semantic equivalence relation w(a i ,a j ) to measure the "semantic correlation" between two answers a i and a j . The form of the calculated semantic equivalence relation matrix W is:
[0049] W = (w i,j )i,j = 1,…,m, i.e.,
[0050]
[0051] Based on this semantic equivalence relation matrix W, the differential entropy of the answer subset X and the answer subset X ′ can be obtained.
[0052] Taking the calculation of the differential entropy of the answer subset X as an example, the present invention first extracts the rows and columns related to the answers in the answer subset X from the semantic equivalence relation matrix W to form a submatrix W x . This submatrix contains the semantic correlation information between all the answers in the answer subset X.
[0053] Based on this submatrix W x , the present invention can calculate the differential entropy of each answer in the answer subset X:
[0054]
[0055] where, {a} is the answer subset containing a single answer a, X is the answer subset not containing the answer a, h(·) is the differential entropy, and v(·) is a function that maps the answer set to an n-dimensional random vector. Calculate v(X):
[0056] v(X) = (s1,g2,···,s k ,g k+1 ,···,g n )
[0057] where, s i is the semantic representation of the answer derived from the semantic equivalence relation matrix W in X, and g i is an independent standard Gaussian variable. h(v(X)) is calculated using the relevant submatrix W x corresponding to the answers in X in W. Where,
[0058]
[0059] det(W x ) represents the determinant of W x .
[0060] Step 2.3: Obtain the Shapley value of the original answer set A based on the Shapley values of all the answers in the original answer set A.
[0061] After obtaining the Shapley value of each answer a the Shapley value of the original answer set A can be obtained:
[0062]
[0063] Step 3: Based on the Shapley value of the original answer set A, determine whether the question is an out-of-distribution sample.
[0064] The present invention sets a threshold based on the expected false positive rate. This threshold is used to determine whether an answer should be accepted or rejected. The false positive rate is a parameter that can be specified by the user, which defines the maximum error proportion that can be tolerated when accepting an answer. If the out-of-distribution score of the answer is less than or equal to the threshold, it is considered that the answer is the output of an in-distribution query, that is, the model has sufficient confidence in this answer and outputs it as the correct answer. If the out-of-distribution score is greater than the threshold, it is considered that the answer is the output of an out-of-distribution query, that is, the model has insufficient confidence in this answer and rejects it.
[0065] In summary, the present invention is specifically designed for large language models in natural language processing tasks to improve their security and reliability. This method does not rely on specific task data and can generalize across tasks. By operating in the semantic event space, it evaluates the semantic consistency of the text, thereby effectively identifying out-of-distribution samples in classification and generation tasks. By sampling multiple sequences from the model prediction distribution and clustering them, constructing a semantic distance matrix, and calculating the out-of-distribution confidence using the Shapley value, combined with the consistency prediction technology to ensure false positive rate control, the performance of the method on various datasets and model scales is significantly improved, especially the accuracy in dealing with far out-of-distribution and near out-of-distribution scenarios.
[0066] The above description only elaborates on a specific example of the present invention and does not impose any restrictions on the present invention. Obviously, for those with professional knowledge in the field, once they understand the content and principle of the present invention, it is possible to make various modifications and changes in form and details without violating the original principle and structure of the present invention. However, these corrections and changes based on the idea of the present invention are still considered to be within the protection scope of the claims of the present invention.
Claims
1. A large-scale language model out-of-distribution sample detection method suitable for general tasks, characterized in that: The method comprises: Generate an original answer set A for the question based on the large language model, wherein the original answer set A includes multiple answers a to the question; Calculating the Shapley value of the original answer set A; Based on the Shapley value of the original answer set A, determine whether the question is an out-of-distribution sample.
2. The method according to claim 1, characterized in that The calculating of the Shapley value of the original answer set A comprises: Select an answer subset X that does not contain answer a from the original answer set A, and add the answer a to the answer subset X to obtain the answer subset X. ′ ; Calculate the answer subset X and the answer subset X respectively ′ The differential entropy of Based on the answer subset X and the answer subset X ′ The differential entropy change of , we get the Shapley value of the answer a According to the Shapley values of all answers in the original answer set A, the Shapley value of the original answer set A is obtained.
3. The method according to claim 2, characterized in that Calculate the differential entropy of the answer subset X, including: Based on the semantic distance between each pair of answers in the original answer set A, a semantic equivalence relationship matrix W is constructed; Extract the rows and columns related to the answer subset X from the semantic equivalence relationship matrix W to form a submatrix W x ; Based on the submatrix W x , calculate the differential entropy of the answer subset X.
4. The method according to claim 2, characterized in that: The Shapley value Where h(v(X)) represents the differential entropy of the answer subset X, v(·) is the function that transforms the answer subset X to an n-dimensional random vector, and h(v(X∪{a})) represents the differential entropy of the answer subset X. ′ The differential entropy of .
5. The method according to claim 1, characterized in that Based on the Shapley value of the original answer set A, judging whether the question is an out-of-distribution sample includes: Obtain a set threshold value and compare it with the Shapley value of the original answer set A; If the Shapley value of the original answer set A is less than the set threshold, it is determined that the question is an in-distribution sample; If the Shapley value of the original answer set A is greater than the set threshold, it is determined that the question is an out-of-distribution sample.
6. A large-scale language model out-of-distribution sample detection system suitable for general tasks, characterized by: The system comprises: A generation module, used for generating an original answer set A of the question based on the large language model, wherein the original answer set A includes multiple answers a of the question; A calculation module, used for calculating the Shapley value of the original answer set A; The detection module is used to determine whether the question is an out-of-distribution sample based on the Shapley value of the original answer set A.
7. An electronic device, characterized in that: The electronic device comprises: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the method for detecting out-of-distribution samples of a large language model suitable for general tasks as described in any one of claims 1 to 5 is implemented.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by the processor, the method for detecting out-of-distribution samples of a large language model suitable for general tasks as described in any one of claims 1 to 5 is implemented.
9. A computer program product, characterized in that When the computer program product runs on a computer device, the computer device executes the large-scale language model out-of-distribution sample detection method applicable to general tasks as described in any one of claims 1 to 5.