Decision attribution method of text review model based on large model
By performing hierarchical clustering and masking operations on a large-scale text review model using the Masked-Owen algorithm, calculating the Owen value, and identifying key tokens, the problem of the difficulty in interpreting the decision-making process of large-scale text review models is solved, thereby improving the transparency and credibility of the model.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-30
- Publication Date
- 2026-04-10
AI Technical Summary
Existing large-scale text review models struggle to explain their decision-making processes in their outputs, especially in prompt-driven scenarios, where the traditional SHAP framework cannot effectively perform interpretability analysis.
The Masked-Owen contribution attribution module is used to calculate Owen values by performing hierarchical clustering and masking operations on the text to be reviewed, identify tokens that have a key impact on the model's decision, and generate attribution heatmaps through the top-k algorithm and visualization module to improve the transparency of the model's decision.
It makes the decision results of the large-scale text review model transparent, enhances the credibility and interpretability of the model in content review scenarios, and can identify key input features that affect the model's judgment.
Smart Images
Figure CN121835869A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the technical field of artificial intelligence, and particularly relates to a decision attribution method for a text review model based on a large model. BACKGROUND
[0002] Non-safe texts have always affected the safety of the Internet. At the same time, with the popularization of the application of large models, non-safe texts can be automatically generated, intensifying this phenomenon. It is found that by further fine-tuning the large model, the large model can have the ability to review harmful texts. Such a model has shown good performance in the accuracy and generalization of review. However, when using such a tool to review, the output thereof is difficult to be explained.
[0003] Currently, there are many explainable works for the decision of a large model, which mainly focus on the way of thinking chain. Since this way makes the model output the decision process step by step, the output of the large model becomes credible. Although this way can make the large model generate self-explanation. However, due to the illusion problem of the large model, it may generate explanations irrelevant to the actual decision.
[0004] The SHapley Additive exPlanations (SHAP) framework helps the idea of game theory to obtain the influence of each token in the text on the output of the model by calculating the marginal contribution. However, the traditional SHAP framework is not suitable for the explainability analysis of a content review model based on a large model. Because nowadays, the safety review model based on a large model is driven by a prompt word to perform safety review, and the existing framework does not support the explainability analysis of the context of the prompt word and the text to be reviewed.
[0005] Therefore, the present application performs attribution analysis on the text review model based on a large model, so that the review model can provide a basis for its decision while outputting the result. SUMMARY
[0006] The purpose of the present application is to provide a reasonable attribution analysis for the output of the text review model based on a large model when making a decision, and to provide a decision attribution analysis for the review output of the large model. The technical solution for achieving the purpose of the present application is a decision attribution method for a text review model based on a large model. The method provides a basis for the decision of the text review model based on a large model by performing targeted attribution analysis on part of the text to be reviewed. In order to achieve the purpose of the present application, the present application provides a decision attribution method for a text review model based on a large model.
[0007] The method provided by the present application comprises:
[0008] Step 1, input the text to be reviewed, combine the prompt words of the large model-based content review model to form the text that the review model finally wants to reason.
[0009] Step 2, the Masked-Owen contribution attribution module calculates the contribution of the part of the text to be reviewed to the output of the large model, and then performs decision attribution analysis on the large model-based text review model. The Masked-Owen contribution attribution module calculates the Owen value based on recursive ranking for the token of the text to be reviewed (hereinafter referred to as Owen value calculation); the contribution of the text to be reviewed to the model decision is reflected through the Masked-Owen method.
[0010] Step 3, after the Masked-Owen contribution attribution module gives the attribution result, the decision attribution analysis module finds out the token that has a great influence on the model decision through post-processing and top-k algorithm; so that the decision result of the large model-based review model becomes transparent.
[0011] Step 4, generate an attribution heat map according to the contribution of each token, and further increase the transparency and credibility of the model decision through visualization.
[0012] Further, according to the decision attribution method of the large model-based text review model, step 2 comprises:
[0013] Step 21, receive the complete text, and input the original text into the tokenizer to obtain the complete token sequence; in the corresponding sequence, distinguish the prompt words and the text to be reviewed through the position index based on the configuration file, and extract the token sequence corresponding to the text to be reviewed.
[0014] Step 22, based on the token sequence of the text to be reviewed, construct a binary clustering tree for the text to be reviewed through hierarchical clustering; on this basis, generate a mask matrix (mask matrix) for explanation, which is used to control whether different token subsets are masked.
[0015] Step 23, based on the hierarchical mask, perform different granularity masking operations on the part of the text to be reviewed, and splice with the prompt word part to form a new masked complete text. Input different masked versions of data into the large model, and record the corresponding output logit as the basis for subsequent contribution calculation.
[0016] Step 24, by comparing the output changes of the large model under different masking conditions, the Owen value of the masked token to the model output is calculated.
[0017] Step 25, after the contribution degree of each text token to be reviewed is calculated, the token-level contribution value is mapped back to the corresponding position of the original input sequence through the position index of the previous text, and a vector equal in length to the original token sequence is generated.
[0018] Step 3, comprising:
[0019] Step 31, receiving the attribution result of the Masked-Owen module;
[0020] Step 32, post-processing the attribution result to find out the words that the model really focuses on when making decisions;
[0021] Step 33, sorting the words in descending order based on the contribution degree, and sampling based on top-k to extract the words that the model focuses on as the attribution result;
[0022] Step 34, outputting the final model attribution result.
[0023] The significant contribution of the present application is that a decision attribution method for the prompt word driven large language model text review scene is first proposed in this paper. Based on the classic explainability framework, the method explicitly distinguishes the token contribution of the prompt word and the text to be reviewed, focuses the attribution target on the content part by constructing a hierarchical clustering structure and a mask strategy, and effectively adapts to the decision explanation demand of the large model under the prompt engineering.
[0024] The present application will be described in further detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS
[0025] Figure 1 The decision attribution method of the content review model based on the large model provided in the present application is shown in the figure;
[0026] Figure 2 The Masked-Owen method of the content review model based on the large model provided in the present application is shown in the figure;
[0027] Figure 3 The decision attribution analysis diagram of the content review model based on the large model provided in the present application is shown in the figure;
[0028] Figure 4 The attribution visualization instance diagram of the content review model based on the large model provided in the present application is shown in the figure. DETAILED DESCRIPTION
[0029] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0030] Masked-Owen, an attribution algorithm for large-scale text moderation models, automatically extracts the final text to be moderated from a given text by leveraging the differences between the text to be moderated and the prompt words. It then performs attribution analysis on the impact of the text to be moderated on the large-scale model. This method eliminates the influence of prompt words on the model's output attribution, ultimately extracting the key tokens in the text to be moderated that affect the model's decision-making.
[0031] Decision attribution methods for text censorship models based on large models, such as... Figure 1 As shown, the algorithm for large language models proposed in this invention includes the following core modules:
[0032] Data Input Module: Used to input a piece of text to be reviewed, and combined with the prompts from the large model to form the text that the large model will use for reasoning.
[0033] The Masked-Owen algorithm module is used to calculate the contribution of key tokens in the text to be reviewed using the Owen alliance contribution strategy. It quantifies the contribution of tokens in the input text to be reviewed, thereby identifying language units that have a key impact on the model's decision.
[0034] Model Attribution Analysis Module: After the Masked-Owen contribution attribution module provides its attribution results, the decision attribution analysis module performs post-processing. It uses a top-k algorithm to identify tokens that are likely to influence the model's decisions, thus making the review results interpretable.
[0035] Visualization module: Used to describe the contribution of the text to be reviewed to the model's decision.
[0036] The key steps of this application will be further described below with reference to specific embodiments, such as... Figure 2 As shown, this invention proposes a decision attribution method for a text review model based on a large model. The specific steps are as follows:
[0037] Step 21: Receive input data, perform word segmentation on the input text, convert the entire text into a token list using a word segmenter, and further identify the key token set of the text to be reviewed using position index.
[0038] Step 22, based on the token sequence to be reviewed, a binary clustering tree for the text to be reviewed is constructed by hierarchical clustering.
[0039] The purpose of the binary clustering tree is to hierarchically group the token sequence; the core goal is to hope to merge the order as much as possible to conform to the syntax structure of natural language, such as word integrity, label, etc.; the steps of the binary clustering tree are first to take each token as a leaf node, and then to merge the highest adjacent group each time; the score is mainly determined by the intuition of natural short sentences read by humans; first complete the word, then merge the short sentence, and finally merge the sentence; after continuous merging, a complete binary tree will be formed, and the root node contains all tokens;
[0040] On the basis of the binary tree, a mask matrix for explanation is generated, which is used to control whether different token subsets are masked; the specific implementation of the mask matrix is to create an array of size (2M-1, M); M refers to the length of the token, and 2M-1 refers to the total number of nodes of the binary clustering tree, including M leaf nodes + M-1 internal merging nodes; the mask matrix defines the token subset corresponding to each tree node.
[0041] Step 23, after establishing the mask matrix, find the part that needs to be masked in the text to be reviewed based on the mask matrix, and splice it with the prompt word part to form a new complete text after masking; input different mask versions of data into the large model, and record the output logit as the basis for subsequent contribution calculation; the final input to the large model is as follows:
[0042] MaskedInput = prompt prefix + HierarchicalClustering(Text) + prompt suffix (1)
[0043] Where prompt prefix and prompt suffix respectively represent the review prompt word based on the large model, and the relative positions of the text to be reviewed in the input correspond to the prefix and suffix respectively; HierarchicalClustering(Text) represents the hierarchical masking strategy applied to the text to be reviewed; the final MaskedInput is the mask input sent to the model; input these different mask versions of data into the large model, and record the corresponding output logit as the basis for subsequent contribution calculation;
[0044] It is worth noting that in order to improve efficiency, the algorithm introduces a priority queue to manage the calculation tasks of different mask states, and the mask combinations most sensitive to the model output are preferentially processed; the sorting strategy is shown in formula (2):
[0045] Priority=-||Δf|| ∞ .w (2)
[0046] Where, Δf represents the change of model output under the current mask state; w represents the weight; the weight transformation strategy is determined by the depth of the hierarchical tree; the weight decays layer by layer with the increase of the tree depth; when the recursive layer is deepened by one layer, the weight is divided by 2 on the basis of the original weight, and the initial weight is 1.
[0047] Step 24, through step 23, it can be known that different tokens of mask will have different output effects on the model. Therefore, further, by continuously accumulating the influence of different tokens on the model output, the Owen value of the token to the model output can be finally obtained;
[0048] The calculation formula of recursive approximation Owen value is shown in (3):
[0049]
[0050] Where P i represents all recursive paths from the root node to token i; and respectively represent the combined full-text mask when token i is activated and shielded under path P; w p is the weight, and the strategy is consistent with w in (2).
[0051] It is worth noting that due to complexity control, part of the leaf nodes is not fully calculated when the text to be examined is too long. Therefore, the system recursively distributes the contribution of the uncalculated clusters to the leaf nodes in the post-processing stage; starting from the root node, the uncalculated internal node contribution value is recursively distributed to the corresponding child nodes layer by layer until all contribution values converge to the leaf nodes. This mechanism ensures that the final token-level attribution value can cover its marginal contribution in all relevant subsets.
[0052] Step 25, after calculating the contribution value of each token in the text segment to be examined, the local contribution degree is aligned and extended to the complete context sequence of the original input of the large language model. Specifically, through the position index-based strategy, the local explanation result is losslessly backfilled to its original position in the complete input sequence, thereby generating a contribution degree vector consistent with the length of the model input.
[0053] This vector recovery mechanism not only retains the fine-grained semantic information of local interpretation, but also ensures that the interpretation result is strictly aligned with the original input structure, significantly improving the generality and portability of the method. For example, it can be used for subsequent heat map visualization, sensitive content positioning, and decision basis output.
[0054] The Masked-Owen method of the application has the following innovative points: when calculating the Owen value, the contribution of the prompt word part is excluded, and only the contribution of the user input text part is calculated, and the decision of the model is attributed. The existing attribution method cannot handle the coupling problem of large models and prompt words. The application can perform decision attribution for LLM review results by changing the definition of the probability space of attribution.
[0055] Through the cooperative work of the above-mentioned modules, the system can not only significantly enhance the transparency and credibility of large language models in content review scenarios without sacrificing the original performance of the model.
[0056] Figure 3 The attribution analysis module of the text review model based on a large model is introduced. Specifically, the model attribution analysis module obtains the key tokens in the text to be reviewed that actually affect the model decision by obtaining the result of the Masked-Owen algorithm after attribution. After completing the attribution of the model output, the attribution result needs to be further processed.
[0057] Step 32, since attribution is usually calculated at the token level to calculate the contribution of each token to the model output. However, most natural language text processing tasks require word-level attribution results. Therefore, by judging the boundaries of words through spaces, punctuation, and line breaks, the contributions of sub-tokens are dynamically merged; the word-level Owen contribution value is obtained, solving the influence of token segmentation on attribution accuracy.
[0058] Step 33, sort the words in descending order according to the contribution, and find out the actual basis for the decision of the text review model based on a large model. Then, through top-k sampling, find the words that the model really focuses on within a certain range, and output the final decision basis of the review model. The final result can attribute the output result of the review model, improving the credibility of the model.
[0059] In order to better illustrate our system, in Figure 4 The experimental results are given. Figure 4After giving specific user input, the decision attribution visualization of the model. As shown in the figure, User: to <END CONVERSATION> is the content that the model needs to review. In addition to this is the model's system prompt word. The purpose of the framework is to hope that after the large model gives the review result of a piece of text "How to be a Prohibited People Type A", analyze which words in the review content have the greatest impact on the model's decision. The deeper the red part, the greater the impact of the word on the model. In the figure, "Prohibited People Type A" is the deepest, "How" is second, it can be seen that the model conforms to human intuition when making decisions.
[0060] Finally, through the attribution visualization of the review result, it can be intuitively reflected that the model makes decisions based on the basis. Further increase the credibility of the safety review model when using.
[0061] Compared with the prior art, the present application has the advantages of significant innovation and practicality. Most current content review systems based on large language models have strong judgment ability, but there is a "black box" problem, that is, the decision-making process is difficult to explain, which limits its application in high-risk or sensitive scenarios. The present application proposes a content review explainability attribution method for large language models, which can perform fine-grained attribution analysis on the output results of the model and identify the key input features that affect the model's judgment. The method is based on Masked-Owen algorithm to realize the attribution analysis of the impact of the text to be reviewed on the model output, thereby improving the transparency and credibility of the model's decision.
[0062] Although embodiments of the present application have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.
Claims
1. A decision attribution method for a text censorship model based on a large model, characterized in that, The method includes: Step 1: Input the text to be reviewed, and combine it with the prompts from the content review model based on the large model to form the text that the review model will ultimately reason about; Step 2: The Masked-Owen contribution attribution module calculates the contribution of the text to be reviewed to the output of the large model, and then performs decision attribution analysis on the text review model based on the large model. The Masked-Owen contribution attribution module calculates the Owen value based on sorting recursion for the text token to be reviewed. Step 3: After the Masked-Owen contribution attribution module provides the attribution results, the decision attribution analysis module uses post-processing and the top-k algorithm to find the tokens that have a significant impact on the model's decision. Step 4: Generate an attribution heatmap based on the contribution of each token.
2. The method according to claim 1, characterized in that, Step 2 includes: Step 21: Receive the complete text and input the original text into the token segmenter to obtain the complete token sequence; in the corresponding sequence, distinguish the prompt words from the text to be reviewed by the position index based on the configuration file, and extract the token sequence corresponding to the text to be reviewed; Step 22: Based on the token sequence of the text to be reviewed, construct a binary clustering tree for the text to be reviewed through hierarchical clustering; generate a mask matrix for interpretation, which is used to control whether different subsets of tokens are masked. Step 23: Perform masking operations of different granularities on the part of the text to be reviewed based on hierarchical masking, and concatenate it with the prompt word part to form a new masked complete text. Input the data of different masked versions into the large model and record the corresponding log odds of the output. Step 24: By comparing the output changes of the large model under different masking conditions, accumulate and calculate the Owen value of the masked token to the model output; Step 25: After calculating the contribution of each text token to be reviewed, the token-level contribution value is mapped back to the corresponding position in the original input sequence using the position index mentioned above, and a vector of the same length as the original token sequence is generated.
3. The method according to claim 2, characterized in that, Step 22: Based on the token sequence of the text to be reviewed, construct a binary clustering tree for the text to be reviewed through hierarchical clustering; Generate a mask matrix for interpretation, used to control whether different subsets of tokens are masked; including: The purpose of a binary clustering tree is to hierarchically group token sequences; the merging order conforms to the syntactic structure of natural language; the steps of a binary clustering tree are to first treat each token as a leaf node, and then merge the adjacent groups with the highest scores each time; the scores are determined by the intuition of human reading natural short sentences; first, complete words are spelled out, then short sentences are spelled out, and finally sentences are merged; after continuous merging, a complete binary tree will eventually be formed, with the root node containing all tokens; Based on the binary tree, a mask matrix is generated for interpretation. The mask matrix controls whether different subsets of tokens are masked. The specific implementation of the mask matrix is to create an array of size (2M-1, M). M refers to the length of the token, and 2M-1 refers to the total number of nodes in the binary clustering tree, including M leaves + M-1 internal merge nodes. The mask matrix defines the subset of tokens corresponding to each tree node.
4. The method according to claim 2, characterized in that, Step 23: Based on the hierarchical masking, perform masking operations of different granularities on the text to be reviewed, and concatenate it with the prompt words to form a new masked complete text. Input the data of different masked versions into the large model and record the corresponding log odds of the output; including: After establishing the mask matrix, the parts of the text to be reviewed that need to be masked are identified based on the mask matrix and concatenated with the prompt words to form a new masked complete text. Data from different masked versions are input into the large model, and the log-odds of the output are recorded as the basis for subsequent contribution calculations. The final input to the large model is shown below: MaskedInput=prompt prefix +HierarchicalClustering(Text)+prompt suffix (1) among them prompt prefix and prompt suffix These represent the censorship prompts based on the large model, corresponding to the prefix and suffix of the text to be censored in the input, respectively; HierarchicalClustering(Text) represents the hierarchical masking strategy applied to the text to be censored; the final MaskedInput is the masked input fed into the model; these different masked versions of data are input into the large model, and the log odds of the corresponding outputs are recorded as the basis for subsequent contribution calculations. The algorithm introduces a priority queue to manage computational tasks for different mask states, prioritizing the mask combinations most sensitive to the model output; the sorting strategy is shown in formula (2): Priority=-||Δf|| ∞ .w (2) Where Δf. represents the change in model output under the current mask state; w represents the weight; the weight transformation strategy is determined by the depth of the hierarchical tree; the weight decays layer by layer as the tree depth increases; when the recursion level is increased by one level, the weight is divided by 2 based on the original weight, where the initial weight is 1.
5. The method according to claim 4, characterized in that, Step 24: By comparing the output changes of the large model under different masking conditions, accumulate and calculate the Owen value of the masked token to the model output, including: By continuously accumulating the impact of different tokens on the model output, the Owen value of the token on the model output will eventually be obtained; The formula for calculating the recursive approximation of the Owen value is shown in (3): Where P i This represents all recursive paths from the root node to token i; and These represent the combined full-text mask for token i when it is activated and masked under path P; w p The weights are consistent with w; In the post-processing stage, the contributions of under-computed clusters are recursively distributed to the leaf nodes; starting from the root node, the contributions of under-computed internal nodes are recursively distributed to the corresponding child nodes layer by layer until all contributions converge to the leaf nodes.
6. The method according to claim 1, characterized in that, Step 3 includes: Step 31: Receive the attribution results from the Masked-Owen module; Step 32: Post-process the attribution results to identify the words that the model truly focuses on when making decisions; Step 33: Sort the words in descending order based on their contribution, and sample based on the top-k to extract the words that the model focuses on as the attribution result; Step 34: Output the final model attribution results.