Lightweight method and system for large model inference based on dictionary-level token mask and medium
By using a dictionary-level token masking mechanism, a theme-specific mask is generated, which solves the problem of high computational overhead in reasoning for large language models. This achieves efficient reasoning speed and improved model interpretability, and is suitable for multiple theme scenarios such as intelligent customer service, text generation, and machine translation.
Patent Information
- Application Number
- CN202610607888.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-05-06
- Publication Date
- 2026-08-25
AI Technical Summary
Existing large language models have high computational overhead during the inference phase and lack a unified dictionary-level masking mechanism and a quantitative relationship between token importance and model performance, which limits their deployment and application in resource-constrained scenarios.
By using a dictionary-level token masking mechanism, the system receives topic-related training datasets, constructs multi-hot vector representations, calculates the marginal contribution value of tokens, generates topic-specific dictionary-level masks, performs token-level filtering, reduces inference computation overhead, and improves model interpretability.
It significantly reduces inference computation overhead by 30%-70%, improves model inference speed by more than 2 times, achieves cross-sample consistency and topic-level attribution analysis, and supports adaptive optimization for multi-topic scenarios.
Smart Images
Figure CN122635529A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of large language model reasoning technology, and more specifically, to a lightweight method, system, and medium for large model reasoning based on dictionary-level token masks. Background Technology
[0002] Large Language Models (LLMs) have demonstrated powerful performance in natural language processing tasks, but their high computational overhead during the inference phase limits their deployment in resource-constrained scenarios. Traditional model compression methods, such as pruning and quantization, mainly focus on parameter-level optimization and fail to fully utilize the sparsity of the input data.
[0003] Existing input processing methods mainly include: (1) sample-level data selection, which improves model performance by screening high-quality training samples, but the granularity is coarse and cannot reduce inference overhead; (2) input-specific attribution methods, such as integral gradient, which can generate importance maps for each input, but lacks cross-sample consistency and has high computational overhead; (3) dynamic token pruning method, which dynamically determines the importance of tokens based on context during inference, but it is difficult to achieve pre-computation optimization and interpretability analysis.
[0004] However, existing methods have significant shortcomings: First, they typically lack a unified masking mechanism at the dictionary level, making it impossible to pre-compute fixed token filtering patterns for specific topics or tasks; second, existing methods fail to establish a quantitative relationship between token importance and model performance, lacking theoretical guidance; and finally, existing dynamic methods still require token importance calculation during the inference phase, failing to fully leverage the advantages of pre-computation. Summary of the Invention
[0005] The purpose of this application is to provide a lightweight method, system, and medium for large model inference based on dictionary-level token masking. Through the dictionary-level masking mechanism, a unified token attribution interpretation for inputs with the same topic is achieved, which significantly reduces inference computation overhead and improves model interpretability.
[0006] This application also provides a lightweight method for large model inference based on dictionary-level token masks, including:
[0007] Receive topic-related training datasets;
[0008] The text in the training dataset is tokenized, mapped to the dictionary space, and multi-hot vector representation information is constructed.
[0009] The marginal contribution value of each token in the dictionary is calculated based on the gradient interaction matrix to quantify the degree of influence of each token on the model performance.
[0010] Tokens are sorted and filtered based on their marginal contribution values to generate a theme-specific dictionary-level mask;
[0011] Input a dictionary-level mask into a large model, perform token-level filtering on the input dictionary-level mask sequence, and output the model response results corresponding to the filtered sequence.
[0012] Optionally, in the lightweight method for large model inference based on dictionary-level token masks described in the embodiments of this application, calculating the marginal contribution value of each token in the dictionary based on the gradient interaction matrix specifically includes:
[0013] Calculate the gradient of each sample in the input embedding layer based on the training dataset, and aggregate them to obtain dictionary-level gradient statistics;
[0014] Based on the diagonal Hessian approximation or Fisher information matrix, calculate the curvature-aware correction information for each token;
[0015] The initial marginal contribution value of each token is calculated based on the marginal contribution formula;
[0016] The initial marginal contribution value is corrected based on the curvature-sensing correction information to obtain the final initial marginal contribution value.
[0017] Optionally, in the lightweight method for large model inference based on dictionary-level token masks described in the embodiments of this application, the first... The formula for the marginal contribution of a token is:
[0018]
[0019] in, Represents Token The average weighted gradient, Represents Token The average diagonal of Hessian.
[0020] Optionally, in the lightweight method for large model inference based on dictionary-level token masks described in the embodiments of this application, generating a topic-specific dictionary-level mask specifically includes:
[0021] Set a token retention budget k, and select the top-k tokens based on their marginal contribution values;
[0022] Construct dictionary-level mask vectors ,in Represents Token Retained Represents Token Masked;
[0023] The dictionary-level mask has topic consistency, meaning that all input samples under the same topic share the same mask pattern.
[0024] Optionally, in the lightweight method for large model inference based on dictionary-level token masks described in the embodiments of this application, inputting the dictionary-level mask into the large model specifically includes:
[0025] Receive the input sequence, tokenize the input sequence, and map it to a dictionary space;
[0026] The input sequence is filtered based on a pre-computed dictionary-level mask, and tokens with a mask value of 1 are retained to obtain the filtered input sequence.
[0027] The filtered input sequence is fed into a large model for forward computation, and the model response is output.
[0028] Optionally, in the lightweight method for large model inference based on dictionary-level token masks described in the embodiments of this application, the large model includes a GPT model, an LLaMA model, a Qwen model, or a large language model based on the Transformer architecture.
[0029] Secondly, embodiments of this application provide a lightweight system for large model inference based on dictionary-level token masks. The system includes a memory and a processor. The memory includes a program for a lightweight method for large model inference based on dictionary-level token masks. When the program for the lightweight method for large model inference based on dictionary-level token masks is executed by the processor, it implements the following steps:
[0030] Receive topic-related training datasets;
[0031] The text in the training dataset is tokenized, mapped to the dictionary space, and multi-hot vector representation information is constructed.
[0032] The marginal contribution value of each token in the dictionary is calculated based on the gradient interaction matrix to quantify the degree of influence of each token on the model performance.
[0033] Tokens are sorted and filtered based on their marginal contribution values to generate a theme-specific dictionary-level mask;
[0034] Input a dictionary-level mask into a large model, perform token-level filtering on the input dictionary-level mask sequence, and output the model response results corresponding to the filtered sequence.
[0035] Optionally, in the lightweight large-model inference system based on dictionary-level token masks described in this application embodiment, calculating the marginal contribution value of each token in the dictionary based on the gradient interaction matrix specifically includes:
[0036] Calculate the gradient of each sample in the input embedding layer based on the training dataset, and aggregate them to obtain dictionary-level gradient statistics.
[0037] Based on the diagonal Hessian approximation or Fisher information matrix, calculate the curvature-aware correction information for each token;
[0038] The initial marginal contribution value of each token is calculated based on the marginal contribution formula;
[0039] The initial marginal contribution value is corrected based on the curvature-sensing correction information to obtain the final initial marginal contribution value.
[0040] Optionally, in the lightweight large model inference system based on dictionary-level token masks described in the embodiments of this application, the first... The formula for the marginal contribution of a token is:
[0041]
[0042] in, Represents Token The average weighted gradient, Represents Token The average diagonal of Hessian.
[0043] Thirdly, embodiments of this application also provide a computer-readable storage medium, which includes a lightweight method program for large model inference based on dictionary-level token masks. When the lightweight method program for large model inference based on dictionary-level token masks is executed by a processor, it implements the steps of the lightweight method for large model inference based on dictionary-level token masks as described in any of the preceding claims.
[0044] As can be seen from the above, the lightweight method, system, and medium for large-scale model inference based on dictionary-level token masks provided in this application embodiment receive a topic-related training dataset; tokenize the text in the training dataset, map it to a dictionary space, and construct multi-hot vector representation information; calculate the marginal contribution value of each token in the dictionary based on the gradient interaction matrix to quantify the degree of influence of each token on model performance; sort and filter the tokens according to the marginal contribution value to generate a topic-specific dictionary-level mask; input the dictionary-level mask into a large model, perform token-level filtering on the input dictionary-level mask sequence, and output the model response result corresponding to the filtered sequence; through the dictionary-level mask mechanism, a unified token attribution explanation for inputs with the same topic is achieved, significantly reducing inference computation overhead and improving model interpretability. Attached Figure Description
[0045] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0046] Figure 1 A flowchart of a lightweight method for large model inference based on dictionary-level token masks provided in this application embodiment;
[0047] Figure 2 A flowchart illustrating the token marginal contribution calculation and mask generation process of a lightweight method for large-model inference based on dictionary-level token masks provided in this application embodiment;
[0048] Figure 3 This diagram illustrates the token filtering during the inference phase of a lightweight inference method for large models based on dictionary-level token masks, as provided in an embodiment of this application. Detailed Implementation
[0049] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.
[0050] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0051] Please refer to Figures 1-3 As shown, this lightweight method for large model inference based on dictionary-level token masks is used in terminal devices. This lightweight method for large model inference based on dictionary-level token masks includes the following steps:
[0052] S101, receives topic-related training datasets;
[0053] It should be noted that this invention assumes that the given dataset contains N samples, each of which is a text sequence. The text is mapped to a dictionary space of size C by a pre-trained tokenizer.
[0054] S102, tokenize the text in the training dataset, map it to the dictionary space, and construct multi-hot vector representation information;
[0055] Specifically, for each sample The token sequence is obtained through a token segmenter, and a multi-hot vector representation is constructed, in which... Represents Token Appeared in the sample middle.
[0056] S103, calculate the marginal contribution value of each token in the dictionary based on the gradient interaction matrix, and quantify the degree of influence of each token on the model performance;
[0057] Specifically, this step is implemented through the following sub-steps:
[0058] S1031, calculate the gradient of each sample in the input embedding layer based on the training dataset. For each sample ( , ), calculate the gradient of the loss function with respect to the input layer output, and then calculate the gradient contribution of each token.
[0059]
[0060] in This represents the sample index in the training dataset; Represents the Token index in the dictionary space; Indicates the model in parameters The following is aimed at the first The loss function value calculated for each sample; For the first in the dictionary The embedding vector corresponding to each token; e The embedding vector corresponding to the mask can be defined as... A zero-dimensional vector with consistent dimensions; Indicates the first In the nth sample The gradient contribution value of each token is used to quantify the degree of influence of that token on the loss function in that sample.
[0061] S1032, aggregate to obtain dictionary-level gradient statistics. For each token Calculate its cumulative gradient across all samples containing that token:
[0062]
[0063] in The total number of samples in the training dataset, For indicator functions, used to represent the first... Does the token appear in the ? In the sample, Represents Token Appeared Middle; otherwise .
[0064] S1033, Calculate the average diagonal Hessian for each token based on the diagonal Hessian approximation or Fisher information matrix. In implementation, the sample average of the gradient outer product can be used as an approximation of Fisher information:
[0065]
[0066] S1034, Calculate the Shapley value for each token according to the marginal contribution formula:
[0067]
[0068] The formula is derived based on standard Taylor expansion, requiring no path integrals or complex game theory calculations, and has a solid mathematical foundation.
[0069] S104, Sort and filter the tokens according to their marginal contribution values to generate a dictionary-level mask specific to the theme;
[0070] Specifically, set a token retention budget k (e.g., k=0.3C, meaning 30% of the tokens are retained), select the top-k tokens with the highest marginal contribution values, and construct a dictionary-level mask:
[0071]
[0072] The dictionary-level mask has topic consistency, meaning that all input samples under the same topic share the same mask pattern. .
[0073] S105: Input the dictionary-level mask into the large model, perform token-level filtering on the input dictionary-level mask sequence, and output the model response result corresponding to the filtered sequence.
[0074] In one specific embodiment of the present invention, a training dataset related to a topic corresponding to a target application scenario is received. The training dataset contains a preset number of text samples, and all text samples revolve around a single target topic. The target application scenario includes, but is not limited to, intelligent customer service, text generation, and machine translation.
[0075] A pre-trained tokenizer is used to tokenize each text sample in the training dataset, decomposing the text sample into several tokens. At the same time, all tokens are mapped to a preset dictionary space, which contains all tokens that may appear under the target topic. The size of the dictionary space is a preset fixed value.
[0076] The marginal contribution value of each token in the dictionary space is calculated based on the gradient interaction matrix. This marginal contribution value is used to quantify the comprehensive impact of each token on the inference performance, inference accuracy and inference efficiency of the large model. The gradient interaction matrix is constructed based on the gradient information of the training dataset and contains the gradient interaction relationship between each token.
[0077] All tokens are sorted and filtered from high to low based on their marginal contribution values. Tokens with marginal contribution values below a preset threshold or those ranked last are removed. A topic-specific dictionary-level mask is generated that uniquely corresponds to the target topic. The dictionary-level mask is a fixed binary vector that ensures that the same token always maintains a consistent mask state (active or masked) in different input samples under the same topic, and that the mask state does not change with the context of the input sample.
[0078] The pre-generated topic-specific dictionary-level mask is embedded into the inference module of the large model. During the inference stage of the large model, the newly received input sequence is filtered at the token level. Only tokens with a mask value of 1 are retained to participate in the forward computation of the large model, while tokens with a mask value of 0 are blocked to avoid invalid tokens occupying computing resources.
[0079] The large model performs forward computation on the filtered token sequence to generate a model response result corresponding to the input sequence. The accuracy of the response result is controlled within a preset range compared to the unfiltered response accuracy, while the inference computation overhead is significantly reduced compared to the unfiltered result.
[0080] According to an embodiment of the present invention, the marginal contribution value of each token in the dictionary is calculated based on the gradient interaction matrix, specifically including:
[0081] Calculate the gradient of each sample in the input embedding layer based on the training dataset, and aggregate them to obtain dictionary-level gradient statistics.
[0082] Based on the diagonal Hessian approximation or Fisher information matrix, calculate the curvature-aware correction information for each token;
[0083] The initial marginal contribution value of each token is calculated based on the marginal contribution formula;
[0084] The initial marginal contribution value is corrected based on the curvature-sensing correction information to obtain the final initial marginal contribution value.
[0085] According to an embodiment of the present invention, the first The formula for the marginal contribution of a token is:
[0086]
[0087] in, Represents Token The average weighted gradient, Represents Token The average diagonal of Hessian.
[0088] Specifically, all text samples in the training dataset are input into the large model, and the gradient information of each sample in the large model's input embedding layer is calculated. The gradient information reflects the degree of influence of each token in the sample on the model's loss function. The larger the absolute value of the gradient, the more significant the influence of the token on the model's loss.
[0089] The gradient information of all samples is aggregated, and the mean gradient, maximum gradient and cumulative gradient of each token in all samples containing that token are calculated to obtain dictionary-level gradient statistics. The dictionary-level gradient statistics are used to characterize the gradient contribution of each token at the level of the entire training dataset.
[0090] The curvature-aware correction term for each token is calculated using either the diagonal Hessian approximation method or the Fisher information matrix method. The curvature-aware correction term is used to correct the bias caused by the first-order gradient and improve the accuracy of the token marginal contribution value calculation. The Fisher information matrix is approximated by the sample average of the gradient outer product.
[0091] Based on a preset marginal contribution formula, combined with dictionary-level gradient statistics and curvature-aware correction terms, the Shapley value of each Token is calculated, and this Shapley value is used as the marginal contribution value of the Token. The marginal contribution formula includes a first-order gradient term, a self-interaction term, and a cross-interaction term, which can comprehensively reflect the importance of the Token itself and the interaction influence between Tokens.
[0092] According to an embodiment of the present invention, generating a topic-specific dictionary-level mask specifically includes:
[0093] Set a token retention budget k, and select the top-k tokens based on their marginal contribution values;
[0094] Construct dictionary-level mask vectors ,in Represents Token Retained Represents Token Masked;
[0095] Dictionary-level masks have topic consistency, meaning that all input samples under the same topic share the same mask pattern.
[0096] According to an embodiment of the present invention, inputting a dictionary-level mask into a large model specifically includes:
[0097] Receive the input sequence, tokenize the input sequence, and map it to a dictionary space;
[0098] The input sequence is filtered based on a pre-computed dictionary-level mask, and tokens with a mask value of 1 are retained to obtain the filtered input sequence.
[0099] The filtered input sequence is fed into a large model for forward computation, and the model response is output.
[0100] According to embodiments of the present invention, the large model includes the GPT model, the LLaMA model, the Qwen model, or a large language model based on the Transformer architecture.
[0101] According to an embodiment of the present invention, the method further includes multi-topic extension:
[0102] For T different topics, learn an independent dictionary-level mask for each topic t. ;
[0103] During the inference phase, the input is first classified into topics using a lightweight topic classifier to determine the topic label t;
[0104] Call the dictionary-level mask for the corresponding topic Perform token filtering.
[0105] In summary, compared with the prior art, the beneficial effects of the technical solution of the present invention are as follows:
[0106] 1. This invention provides a lightweight inference method for large models based on dictionary-level token masks. Current sample-level data selection methods, such as LESS, cannot reduce inference overhead, while dynamic token pruning methods lack pre-computation optimization. This invention calculates the marginal contribution value of tokens through gradient interaction matrices to generate topic-specific dictionary-level masks, achieving pre-computation filtering during the inference stage. This can reduce sequence length by 30%-70% and significantly reduce attention computation overhead. This achieves a more than 2-fold increase in reasoning speed;
[0107] 2. This invention combines the marginal contribution analysis of Shapley values to provide a theoretical quantification of token importance. Unlike heuristic-based token pruning methods, the marginal contribution formula of this invention is based on a second-order Taylor expansion, which has a solid mathematical foundation and strong interpretability.
[0108] 3. This invention implements a dictionary-level unified mask, allowing inputs with the same topic to share the same token filtering pattern. Compared to input-specific attribution methods, the dictionary-level mask of this invention has cross-sample consistency, facilitating topic-level attribution analysis and model debugging;
[0109] 4. This invention supports multiple theme scenarios and can learn exclusive masks for different themes, achieving fine-grained theme adaptive inference optimization.
[0110] Secondly, embodiments of this application provide a lightweight system for large model inference based on dictionary-level token masks. The system includes a memory and a processor. The memory includes a program for a lightweight method for large model inference based on dictionary-level token masks. When the program for the lightweight method for large model inference based on dictionary-level token masks is executed by the processor, it implements the following steps:
[0111] Receive topic-related training datasets;
[0112] The text in the training dataset is tokenized, mapped to the dictionary space, and multi-hot vector representation information is constructed.
[0113] The marginal contribution value of each token in the dictionary is calculated based on the gradient interaction matrix to quantify the degree of influence of each token on the model performance.
[0114] Tokens are sorted and filtered based on their marginal contribution values to generate a theme-specific dictionary-level mask;
[0115] Input a dictionary-level mask into a large model, perform token-level filtering on the input dictionary-level mask sequence, and output the model response results corresponding to the filtered sequence.
[0116] It should be noted that this invention assumes that the given dataset contains N samples, each of which is a text sequence. The text is mapped to a dictionary space of size C by a pre-trained tokenizer.
[0117] Calculate the gradient of each sample in the input embedding layer based on the training dataset. For each sample ( , ), calculate the gradient of the loss function with respect to the input layer output, and then calculate the gradient contribution of each token.
[0118]
[0119] in For the loss of this sample, e is the embedding vector of the token in the dictionary; The embedding vector corresponding to the mask can be defined as... A zero-dimensional vector with consistent dimensions.
[0120] Aggregation yields dictionary-level gradient statistics. For each token... Calculate its cumulative gradient across all samples containing that token:
[0121]
[0122] in, Represents Token Appeared Middle; otherwise .
[0123] Calculate the average diagonal Hessian for each token based on the diagonal Hessian approximation or Fisher information matrix. In implementation, the sample average of the gradient outer product can be used as an approximation of Fisher information:
[0124]
[0125] Calculate the Shapley value for each token using the marginal contribution formula:
[0126]
[0127] The formula is derived based on standard Taylor expansion, requiring no path integrals or complex game theory calculations, and has a solid mathematical foundation.
[0128] According to an embodiment of the present invention, the marginal contribution value of each token in the dictionary is calculated based on the gradient interaction matrix, specifically including:
[0129] Calculate the gradient of each sample in the input embedding layer based on the training dataset, and aggregate them to obtain dictionary-level gradient statistics.
[0130] Based on the diagonal Hessian approximation or Fisher information matrix, calculate the curvature-aware correction information for each token;
[0131] The initial marginal contribution value of each token is calculated based on the marginal contribution formula;
[0132] The initial marginal contribution value is corrected based on the curvature-sensing correction information to obtain the final initial marginal contribution value.
[0133] It should be noted that all text samples in the training dataset are input into the large model, and the gradient information of each sample in the large model's input embedding layer is calculated. The gradient information reflects the degree of influence of each token in the sample on the model's loss function. The larger the absolute value of the gradient, the more significant the influence of the token on the model's loss.
[0134] The gradient information of all samples is aggregated, and the mean gradient, maximum gradient and cumulative gradient of each token in all samples containing that token are calculated to obtain dictionary-level gradient statistics. The dictionary-level gradient statistics are used to characterize the gradient contribution of each token at the level of the entire training dataset.
[0135] The curvature-aware correction term for each token is calculated using either the diagonal Hessian approximation method or the Fisher information matrix method. The curvature-aware correction term is used to correct the bias caused by the first-order gradient and improve the accuracy of the token marginal contribution value calculation. The Fisher information matrix is approximated by the sample average of the gradient outer product.
[0136] Based on a preset marginal contribution formula, combined with dictionary-level gradient statistics and curvature-aware correction terms, the Shapley value of each Token is calculated, and this Shapley value is used as the marginal contribution value of the Token. The marginal contribution formula includes a first-order gradient term, a self-interaction term, and a cross-interaction term, which can comprehensively reflect the importance of the Token itself and the interaction influence between Tokens.
[0137] According to an embodiment of the present invention, the first The formula for the marginal contribution of a token is:
[0138]
[0139] in, Represents Token The average weighted gradient, Represents Token The average diagonal of Hessian.
[0140] According to an embodiment of the present invention, generating a topic-specific dictionary-level mask specifically includes:
[0141] Set a token retention budget k, and select the top-k tokens based on their marginal contribution values;
[0142] Construct dictionary-level mask vectors ,in Represents Token Retained Represents Token Masked;
[0143] Dictionary-level masks have topic consistency, meaning that all input samples under the same topic share the same mask pattern.
[0144] According to an embodiment of the present invention, inputting a dictionary-level mask into a large model specifically includes:
[0145] Receive the input sequence, tokenize the input sequence, and map it to a dictionary space;
[0146] The input sequence is filtered based on a pre-computed dictionary-level mask, and tokens with a mask value of 1 are retained to obtain the filtered input sequence.
[0147] The filtered input sequence is fed into a large model for forward computation, and the model response is output.
[0148] According to embodiments of the present invention, the large model includes the GPT model, the LLaMA model, the Qwen model, or a large language model based on the Transformer architecture.
[0149] According to an embodiment of the present invention, the method further includes multi-topic extension:
[0150] For T different topics, learn an independent dictionary-level mask for each topic t. ;
[0151] During the inference phase, the input is first classified into topics using a lightweight topic classifier to determine the topic label t;
[0152] Call the dictionary-level mask for the corresponding topic Perform token filtering.
[0153] A third aspect of the present invention provides a computer-readable storage medium including a large model inference lightweight method program based on dictionary-level token masking. When the large model inference lightweight method program based on dictionary-level token masking is executed by a processor, it implements the steps of the large model inference lightweight method based on dictionary-level token masking as described above.
[0154] This invention discloses a lightweight method, system, and medium for large-scale model inference based on dictionary-level token masks. The method involves receiving a topic-related training dataset; tokenizing the text in the training dataset and mapping it to a dictionary space to construct multi-hot vector representations; calculating the marginal contribution value of each token in the dictionary based on the gradient interaction matrix to quantify the impact of each token on model performance; sorting and filtering the tokens according to their marginal contribution values to generate a topic-specific dictionary-level mask; inputting the dictionary-level mask into a large model; performing token-level filtering on the input dictionary-level mask sequence; and outputting the model response results corresponding to the filtered sequence. Through the dictionary-level mask mechanism, a unified token attribution explanation for inputs within the same topic is achieved, significantly reducing inference computational overhead and improving model interpretability.
[0155] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.
[0156] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.
[0157] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.
[0158] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0159] Alternatively, if the integrated units of the present invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.
Claims
1. A lightweight method for large-scale model inference based on dictionary-level token masks, characterized in that, include: Receive topic-related training datasets; The text in the training dataset is tokenized, mapped to the dictionary space, and multi-hot vector representation information is constructed. The marginal contribution value of each token in the dictionary is calculated based on the gradient interaction matrix to quantify the degree of influence of each token on the model performance. Tokens are sorted and filtered based on their marginal contribution values to generate a theme-specific dictionary-level mask; Input a dictionary-level mask into a large model, perform token-level filtering on the input dictionary-level mask sequence, and output the model response results corresponding to the filtered sequence.
2. The lightweight method for large model inference based on dictionary-level token masking according to claim 1, characterized in that, The marginal contribution value of each token in the dictionary is calculated based on the gradient interaction matrix, specifically including: Calculate the gradient of each sample in the input embedding layer based on the training dataset, and aggregate them to obtain dictionary-level gradient statistics. Based on the diagonal Hessian approximation or Fisher information matrix, calculate the curvature-aware correction information for each token; The initial marginal contribution value of each token is calculated based on the marginal contribution formula; The initial marginal contribution value is corrected based on the curvature-sensing correction information to obtain the final initial marginal contribution value.
3. The lightweight method for large model inference based on dictionary-level token masking according to claim 2, characterized in that, The first The formula for the marginal contribution of a token is: ； in, Represents Token The average weighted gradient, Represents Token The average diagonal of Hessian.
4. The lightweight method for large model inference based on dictionary-level token masking according to claim 3, characterized in that, Generate a theme-specific dictionary-level mask, specifically including: Set a token retention budget k, and select the top-k tokens based on their marginal contribution values; Construct dictionary-level mask vectors ,in Represents Token Retained Represents Token Masked; The dictionary-level mask has topic consistency, meaning that all input samples under the same topic share the same mask pattern.
5. The lightweight method for large model inference based on dictionary-level token masking according to claim 4, characterized in that, Inputting dictionary-level masks into a large model specifically includes: Receive the input sequence, tokenize the input sequence, and map it to a dictionary space; The input sequence is filtered based on a pre-computed dictionary-level mask, and tokens with a mask value of 1 are retained to obtain the filtered input sequence. The filtered input sequence is fed into a large model for forward computation, and the model response is output.
6. The lightweight method for large model inference based on dictionary-level token masking according to claim 5, characterized in that, The large models include GPT models, LLaMA models, Qwen models, or large language models based on the Transformer architecture.
7. A lightweight system for large-scale model inference based on dictionary-level token masks, characterized in that, The system includes a memory and a processor. The memory contains a program for a lightweight method for large model inference based on dictionary-level token masks. When the program for the lightweight method for large model inference based on dictionary-level token masks is executed by the processor, it performs the following steps: Receive topic-related training datasets; The text in the training dataset is tokenized, mapped to the dictionary space, and multi-hot vector representation information is constructed. The marginal contribution value of each token in the dictionary is calculated based on the gradient interaction matrix to quantify the degree of influence of each token on the model performance. Tokens are sorted and filtered based on their marginal contribution values to generate a theme-specific dictionary-level mask; Input a dictionary-level mask into a large model, perform token-level filtering on the input dictionary-level mask sequence, and output the model response results corresponding to the filtered sequence.
8. The lightweight system for large model inference based on dictionary-level token masking according to claim 7, characterized in that, The marginal contribution value of each token in the dictionary is calculated based on the gradient interaction matrix, specifically including: Calculate the gradient of each sample in the input embedding layer based on the training dataset, and aggregate them to obtain dictionary-level gradient statistics. Based on the diagonal Hessian approximation or Fisher information matrix, calculate the curvature-aware correction information for each token; The initial marginal contribution value of each token is calculated based on the marginal contribution formula; The initial marginal contribution value is corrected based on the curvature-sensing correction information to obtain the final initial marginal contribution value.
9. The lightweight system for large model inference based on dictionary-level token masking according to claim 8, characterized in that, The first The formula for the marginal contribution of a token is: ； in, Represents Token The average weighted gradient, Represents Token The average diagonal of Hessian.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a lightweight method program for large model inference based on dictionary-level token masks. When the lightweight method program for large model inference based on dictionary-level token masks is executed by a processor, it implements the steps of the lightweight method for large model inference based on dictionary-level token masks as described in any one of claims 1 to 6.