Transform-based efficient and safe reasoning method and system for natural language generation model

Abstract

The invention relates to the technical field of natural language generation, in particular to an efficient and safe reasoning method and system for a natural language generation model based on Transform, and the method comprises the steps: inputting a general corpus data set into a pre-training generation model in an offline stage, calculating the importance of each attention head of the model on the general corpus data set; pruning operation is carried out on the attention head of the model based on importance, a pruned generation model is obtained, and Transform is adopted in the generation model; in the online stage, to-be-processed text data is input into the pruned generation model, the importance of Tokens is evaluated layer by layer by utilizing a composite evaluation index, the Tokens are pruned by utilizing a pruning threshold based on the importance of the Tokens, a generation sequence of the to-be-processed text data is obtained through forward propagation of the remaining Tokens, and the to-be-processed text data is obtained. The composite evaluation index is obtained by weighting the importance of a model attention head and a Token attention accumulation score in an off-line stage. According to the method, the safety reasoning efficiency of the natural language generation task is improved through an offline-online collaborative optimization framework.

Description

Technical Field

[0001] This invention relates to the field of natural language generation technology, and in particular to an efficient and secure reasoning method and system for a natural language generation model based on Transformer. Background Technology

[0002] In recent years, large language models (hereinafter referred to as large models) have been widely studied and applied in academia and industry due to their powerful capabilities in natural language understanding and text generation. However, as large models become more popular, the privacy risks associated with their use are becoming increasingly prominent. In practical applications, the common practice is to deploy deep learning models on cloud servers. While this provides convenience for users, it also requires users to upload their data to the cloud. In daily use, personal data contains a large amount of private information and even confidential files, and the fact that the data and model do not belong to the same party poses a significant risk of privacy leakage. Secure Multi-Party Computation (MPC) technology enables different participants to complete data computations while maintaining the privacy of each party's data. Combined with deep learning models, it can meet the data privacy protection requirements of the deep learning-as-a-service model. However, the secure inference process of deep learning models based on MPC has high latency. Therefore, researchers have conducted extensive research on how to efficiently achieve secure inference of deep learning models. Currently, there are many research results on secure inference for the Transformer model, the underlying architecture of large models.

[0003] Research on optimizing Transformer-based secure inference models primarily focuses on computationally expensive operations such as the Softmax attention mechanism and activation functions, with applications mainly targeting Natural Language Understanding (NLU) scenarios. For example, existing research often uses Transformer encoders like BERT, and the commonly used GLUE dataset covers various tasks including sentiment analysis, semantic similarity assessment, and textual entailment, aiming to evaluate the model's NLU capabilities. While Natural Language Generation (NLG) models also use the Transformer architecture, research indicates that optimization methods for secure inference models targeting NLU scenarios do not significantly improve efficiency in NLG tasks. This paper analyzes the secure inference overhead of two typical models, BERT_base and GPT-2, for these scenarios. The analysis shows that in the BERT_base model, the Softmax operation of the attention mechanism and the GeLU activation operation of the hidden layers account for over 80% of the total inference latency. Therefore, efficiency optimization for these two operators is significantly effective in BERT-type secure inference models. In the GPT model, the latency of the above two operators accounts for only 30% of the total inference latency, while linear operations and input / output token processing are the main sources of overhead.

[0004] The main reason for the change in overhead lies in the autoregressive process in the NLG task. In this process, the model generates new tokens one by one and uses all generated tokens as input to predict subsequent content. As the length of the input tokens increases, the processing overhead of the token representation vectors at each layer of the model also increases. Although researchers have significantly reduced the overhead of some repetitive computations through mechanisms such as key-value caching, some non-reusable computations gradually accumulate overhead during the iterative autoregressive process, becoming a bottleneck in the secure inference process of natural language generation models. To address the overhead problem in the autoregressive process, some researchers have proposed pruning the token representation vectors of intermediate layers in the model's inference process to reduce the accumulated overhead of subsequent inference processes and improve the inference efficiency of the model in the NLG task. This optimization method has been studied and applied in plaintext inference models, but in secure inference scenarios, security evaluation needs to be performed on the token vectors of intermediate layers, which introduces new security computation overhead. How to reduce the additional evaluation overhead is a key problem that needs to be solved to optimize the secure inference process of the NLG task based on the token pruning method. Summary of the Invention

[0005] To address the limited improvement in model inference optimization efficiency in existing natural language generation tasks, this invention provides an efficient and secure inference method and system for natural language generation models based on Transformer. It improves the secure inference efficiency of natural language generation tasks through an offline-online collaborative optimization framework.

[0006] According to the design scheme provided by this invention, on the one hand, an efficient and secure inference method for a natural language generation model based on Transformer is provided, comprising:

[0007] In the offline phase, a general corpus dataset is input into the pre-trained generative model. The importance of each attention head of the model is calculated on the general corpus dataset. Based on the importance, the attention heads of the model are pruned to obtain the pruned generative model. The generative model adopts Transformer.

[0008] In the online phase, the text data to be processed is input into the pruned generative model. The importance of the tokens is evaluated layer by layer using a composite evaluation index. Based on the token importance, the tokens are pruned using a pruning threshold. The generated sequence of the text data to be processed is obtained through forward propagation of the remaining tokens. The composite evaluation index is obtained by weighting the importance of the model attention head and the cumulative score of token attention in the offline phase.

[0009] As an efficient and secure inference method for the Transformer-based natural language generation model of this invention, the importance of each attention head of the model is further calculated on a general corpus dataset, including:

[0010] The integral gradient of each attention head in the model is calculated based on the text data in the general corpus dataset, so as to obtain the attention attribution score of the attention head based on the integral gradient;

[0011] Based on the attention attribution score, the expectation of each attention head in the autoregressive process on the general corpus dataset is obtained, and the importance of each attention head on the general corpus dataset is determined based on the expectation.

[0012] As an efficient and secure inference method for the natural language generation model based on Transformer in this invention, the process of obtaining the attention attribution score of the attention head based on the integral gradient is further expressed as follows: Among them, A h Let λ be the attention weight matrix of the attention head h, λ be the step size, F(x) be the function expression of the neural network model, and k be the path parameter in the integral gradient.

[0013] As an efficient and secure reasoning method for the Transformer-based natural language generation model of this invention, the process of determining the importance of each attention head on a general corpus dataset based on expectations is further represented as follows: Among them, I h The attention head h represents the importance of the attention head h on the dataset D, T represents the maximum length of the generated token sequence, and E represents the importance of the attention head h on the dataset D. x～D [·] represents the expectation on dataset D. Let represent the attribution score of attention head h during the generation of the t-th token, where x is the data in dataset D.

[0014] As an efficient and secure inference method for the natural language generation model based on Transformer in this invention, further, a pruning operation is performed on the model attention heads based on importance, including:

[0015] Set pruning thresholds based on the pruning rate;

[0016] If the importance of an attention head on the general corpus dataset is greater than the pruning threshold, the attention head is retained and its importance score is recorded; if the importance of an attention head on the general corpus dataset is less than the pruning threshold, the attention head weight is reset to zero.

[0017] The pruned generative model is obtained based on the retained attention heads, and a scoring matrix is ​​composed of the importance scores of each attention head.

[0018] As an efficient and secure reasoning method for the natural language generation model based on Transformer in this invention, the calculation process of the composite evaluation index is further expressed as follows: Among them, I comb [i] represents the composite importance of the i-th token. The value is a normalized representation of the importance of attention heads h acquired offline, where α is the balance coefficient, n represents the number of tokens, and H is the number of attention heads used to evaluate token importance. h [j,i] represents the correlation between the i-th and j-th tokens in the attention head h.

[0019] As an efficient and secure inference method for natural language generation models based on Transformer in this invention, the secure inference is further based on the Crypton framework to encode each private data in the natural language generation task through secret sharing among multiple participants. The input data is split into multiple fragments and the fragments are secretly shared among different participants to perform predictive inference. Furthermore, a secure comparison protocol is used to perform pruning operations on the tokens. The pruning threshold in the token pruning operation is based on the model loss function and the optimal value is obtained by model fine-tuning.

[0020] Furthermore, this invention also provides an efficient and secure inference system for a natural language generation model based on Transformer, comprising: an offline processing module and an online inference module, wherein,

[0021] The offline processing module is used to input a general corpus dataset into the pre-trained generative model in the offline stage, calculate the importance of each attention head of the model on the general corpus dataset, and perform pruning operation on the attention heads of the model based on the importance to obtain the pruned generative model. The generative model adopts Transformer.

[0022] The online inference module is used to input the text data to be processed into the pruned generative model in the online stage. It evaluates the importance of tokens layer by layer using a composite evaluation index, prunes tokens based on token importance and using a pruning threshold, and obtains the generation sequence of the text data to be processed through forward propagation of the remaining tokens. The composite evaluation index is obtained by weighting the importance of the model attention head and the cumulative score of token attention in the offline stage.

[0023] The beneficial effects of this invention are:

[0024] 1. This invention improves the efficiency of secure inference in natural language generation tasks through an offline-online collaborative optimization framework. By transferring the evaluation of the importance of attention heads related to general language capabilities to the offline stage, the online stage directly performs attention head compression based on the offline evaluation results. Based on the token importance metric compatible with additive secret sharing, the computational burden of online evaluation is significantly reduced, and the evaluation overhead is significantly optimized.

[0025] 2. This invention achieves 40%-60% effective pruning of attention heads and 30% efficient pruning of tokens through a composite importance assessment mechanism, resulting in an overall improvement of 1.69-2.15 times in the model's secure inference efficiency, while maintaining no significant decline in model generation performance. This effectively solves the problem of existing methods struggling to balance model performance and inference efficiency in secure inference environments. It not only significantly reduces the computational load and latency of natural language generation tasks but also greatly reduces the additional overhead introduced by online optimization, providing effective technical support for the practical application of secure inference in natural language generation scenarios. Attached image description:

[0026] Figure 1 This example illustrates the efficient and secure inference process of a Transformer-based natural language generation model.

[0027] Figure 2 This is a schematic diagram of the offline-online collaborative NLG secure inference model optimization framework in the embodiment;

[0028] Figure 3 This example illustrates a comparison of offline attention head pruning effects for models of different sizes.

[0029] Figure 4 This example illustrates the comparison of the impact of offline attention head pruning on model performance before and after fine-tuning.

[0030] Figure 5 This example illustrates a comparison of online token pruning effects under different input lengths.

[0031] Figure 6 This is a diagram illustrating the comparison of the improvement in secure inference efficiency under different input lengths in the embodiments;

[0032] Figure 7 This is a schematic diagram illustrating the comprehensive evaluation results across multiple indicators in the embodiment. Detailed implementation method:

[0033] To make the objectives, technical solutions, and advantages of this invention clearer and more understandable, the invention will be further described in detail below with reference to the accompanying drawings and technical solutions.

[0034] While existing NLG models for natural language generation tasks are also based on the Transformer architecture, research has shown that optimization methods for secure inference models designed for natural language understanding scenarios do not significantly improve efficiency in natural language generation tasks. From the perspective of existing secure inference model optimization methods, natural language generation tasks typically require online completion based on user input. Traditional pruning methods for evaluating model structure are usually performed online, introducing significant evaluation overhead in secure inference environments. Therefore, developing better evaluation methods for pruning objects such as attention heads and tokens to achieve a better balance between model performance and inference efficiency, while minimizing evaluation overhead, is a key issue that needs to be addressed in optimizing secure inference models for NLG tasks related to natural language generation.

[0035] Therefore, in the embodiments of the present invention, see... Figure 1 As shown, this paper presents an efficient and secure inference method for natural language generation models based on Transformer, which specifically includes the following:

[0036] S101. In the offline stage, a general corpus dataset is input into the pre-trained generative model. The importance of each attention head of the model is calculated on the general corpus dataset. Based on the importance, the attention heads of the model are pruned to obtain the pruned generative model. The generative model adopts Transformer.

[0037] S102. In the online phase, the text data to be processed is input into the pruned generative model. The importance of the tokens is evaluated layer by layer using a composite evaluation index. Based on the importance of the tokens and using a pruning threshold, the tokens are pruned. The generation sequence of the text data to be processed is obtained through forward propagation of the remaining tokens. The composite evaluation index is obtained by weighting the importance of the model attention head and the cumulative score of the token attention in the offline phase.

[0038] In natural language generation (NLG) tasks, given an initial input of text data to be processed (such as a prompt, question, or preceding text), a NLG model is used to generate a reasonable, coherent, and compliant sequence of subsequent text. From the perspective of model task execution, the capabilities of an NLG model can be broken down into two parts: general language understanding and language generation. General language understanding is highly task-independent and can be obtained through pre-training on a large-scale general corpus. Accordingly, in this embodiment, the secure inference process of the NLG model is deconstructed, and an offline-online collaborative model optimization framework is used, such as... Figure 2 As shown, firstly, in the offline stage, a general task is used to pre-prune the attention heads in the model. This enables privacy protection scenarios without the need for privacy data, effectively inheriting the advantages of existing advanced attention head pruning methods. Subsequently, in the generation task stage, the cumulative attention score of the token is combined with the importance weights of the attention heads obtained offline to calculate the comprehensive importance score of the token. This optimizes the token importance assessment effect while significantly reducing the overhead of online security assessment, thereby achieving efficient compression of the NLG security inference model.

[0039] Offline Stage: The advantage of this stage is that it does not require optimization of the model based on sensitive inference data and can rely on a wider range of publicly available data and computing resources. However, the key issues to be addressed in this stage are: first, how to decouple from the online generation process while acquiring correlation information with the online input data offline; and second, how to accurately assess the importance of the attention head to compress the model while maintaining its performance as much as possible. For the first issue, the solution lies in decoupling the natural language generation task. Natural language is composed of a finite number of lexical units combined in different orders. This combination relationship has both certain universality, such as basic lexical unit usage and grammatical rules, and its own particularity, such as the same lexical unit in different positions in a sentence expressing completely opposite meanings. Based on these characteristics and the basic principles of the model's attention mechanism, using a general dataset, the ability of the generative model's attention head in general natural language processing is decoupled. This ability also has strong transferability to the online generation process, thus effectively solving the first issue.

[0040] Assessing the importance of attention heads is essentially an attribution problem, which involves quantifying the contribution of internal model components (such as attention heads) to the final inference result to explain the basis of model decisions. Traditional attribution methods mainly include those based on precision differences and those based on Taylor expansion. The basic idea of ​​the precision difference-based method is to successively disable the output of each attention head and then observe the changes in model performance, judging its importance by the impact of different attention heads on performance. This method is simple and intuitive, and directly related to task performance, but it requires multiple inference processes, and the resulting importance depends on the specific task, leading to weak generality. The Taylor expansion-based method mainly utilizes the gradient information of the attention head output in the model's loss function, and evaluates the impact of removing the head on the loss value through first-order or second-order Taylor expansion. This method only requires one forward and backward propagation, making the evaluation more efficient, and it is not dependent on specific task performance indicators, making it more general; however, this method suffers from evaluation instability, and the evaluation error increases significantly when the gradient value is small in the saturation region. To address the problems of existing evaluation methods, and considering the continuity and gradient differentiability of generative tasks, this embodiment employs an Integrated Gradients (IG) method to evaluate the importance of attention heads in the model. Its basic definition is: Taking a straight path from a baseline value x′ to the true input value x, calculate the gradient at each point along the path. The integral of the gradients at each point along this path is the integrated gradient of the input. Its calculation expression is as follows:

[0041]

[0042] Here, F(x) is the functional expression of the neural network model, calculated by integrating the gradients at each point along the input path. A larger integral gradient indicates that the direction of gradient change along the integration path is more consistent with the change in model output, meaning a greater contribution of the input to the model's prediction. This method can solve the gradient saturation problem that may exist with the Taylor expansion method, and it does not depend on specific task labels when evaluating contributions, resulting in highly interpretable evaluation results. Therefore, using the integral gradient method to evaluate the importance of attention heads in generative models is more in line with the needs of the offline stage.

[0043] Online Phase: The key issues to be addressed in this phase are balancing accuracy and efficiency in token pruning and the overhead of token evaluation in secure inference scenarios. For token importance evaluation, existing research has found that effective methods use self-attention weights to evaluate the impact of each token on other tokens. This method can obtain the correlation between token representation vectors, but during pruning, it requires measuring token importance at each layer or module, resulting in weak globality. Furthermore, this method is only input-dependent, and different inputs significantly impact the effectiveness of token pruning. Therefore, this embodiment proposes a composite token importance evaluation method that incorporates the importance of attention heads obtained in the offline phase into the token importance evaluation process. This increases the global attention given to token evaluation through the global importance of attention heads, enabling relative importance measurement between tokens across layers; it also leverages the advantages of attention heads in general language capabilities to improve the stability of online token pruning.

[0044] To address the overhead of token evaluation in secure inference scenarios, the importance weights of the attention heads acquired in the offline stage are incorporated as part of the model parameters and introduced into the online stage. The online token importance evaluation process, based on traditional methods, only adds a weighting operation, which is highly compatible with MPC environments and remains efficient in secure inference scenarios. Therefore, the proposed composite importance evaluation method for the online stage does not introduce additional evaluation overhead on top of existing methods. Furthermore, the model executing the online generation task is a pre-pruned model, and the online evaluation overhead based on model attention weights is also compressed accordingly, thereby reducing the token evaluation overhead in secure inference scenarios.

[0045] Specifically, in the offline stage, the integral gradient of each attention head in the model is first calculated based on the basic principle of integral gradient. Then, the importance attribution score of each attention head is defined and calculated, thereby designing the attention head pruning algorithm for the offline stage.

[0046] In NLG models, the key mechanism of the attention head lies in expressing the interaction relationships between different lexical units through the attention weights between elements. These relationships collectively determine the next generated lexical unit. Therefore, in NLG models, the self-attention attribution score can be used as the criterion for determining the importance of the attention head.

[0047] The self-attention attribution score is calculated using integral gradient. According to expression (1), the weight baseline is set to 0, then the self-attention weight a ij The integral gradient expression can be represented as:

[0048]

[0049] For the attention head h, its attention weight matrix is ​​represented as A. h Attention attribution score based on integral gradient is denoted as Attr. h Then the calculation formula can be expressed as:

[0050]

[0051] For integral operations, a common practical solution is to use Riemann summation for effective approximation. Specifically, the integration path is decomposed into sufficiently small intervals, and the gradient values ​​at each interval are summed. Thus, formula (3) can be simplified to:

[0052]

[0053] Where λ is the step size, which can be set according to actual needs. The attribution score of each attention weight matrix can be calculated according to formula (4).

[0054] To evaluate the importance of each attention head in the NLG model, it is necessary to consider both the attribution score of each attention head when predicting the next token and the dynamic changes in the importance of attention heads during the autoregression process. Therefore, for a single token prediction process, the maximum value is used to capture the most interpretable interaction among each attention head, calculated as follows:

[0055]

[0056] in This indicates that the input to the generative model is x. i The maximum value of the importance attribution of attention head h during the generation of the t-th token represents the importance of the attention head in a single token prediction process. For the autoregressive process, the average cumulative importance of each attention head is calculated to cover the dynamic generation process. The calculation formula is as follows:

[0057]

[0058] In equation (6), T represents the maximum length of the generated sequence for the input data x. i I h (x i The importance of attention head h on the input is represented by ). Based on the goals of the offline phase, importance scores for each attention head are obtained on a general dataset. Let the selected general dataset be D, then the importance I of attention head h on dataset D is... h It can be represented as:

[0059]

[0060] E x～D[·] represents the expectation on dataset D, highlighting the task-independent nature of attention head evaluation by calculating the importance of each attention head on a general corpus.

[0061] Based on the above method for measuring the importance of attention heads, attention heads in the NLG model are pruned in the offline stage. The specific pruning steps are shown in Algorithm 1.

[0062]

[0063]

[0064] Based on the above pruning algorithm, the step size λ for calculating the attention attribution score and the attention head pruning rate ρ need to be input. Regarding the step size setting, experiments show that a value between 20 and 300 is sufficient for approximate calculation of the integral within a 5% deviation. Considering approximation accuracy and computational cost, the step size λ can be uniformly set to 100. The attention head pruning rate ρ needs to be dynamically set based on test results of the impact of different pruning rates on the model's generation capability on a general dataset.

[0065] Online token evaluation needs to be performed in a secure inference environment, which introduces additional ciphertext evaluation overhead. Furthermore, the limitations imposed by the secure inference environment on evaluation methods make it difficult to balance pruning magnitude with model performance. Therefore, based on offline pre-pruning of NLG models, this paper further optimizes the online token evaluation method, proposing an online token pruning method based on composite importance evaluation. By combining attention head importance scores with a composite token importance evaluation metric, this method improves upon the traditional token importance evaluation method based solely on attention weights, enabling the evaluation of different dimensions of tokens. The attention weights are primarily used to evaluate the internal correlation between tokens, while the scores of each attention head represent the general understanding ability acquired offline. Both together determine the importance of a token. This combination of dynamic and static approaches better preserves the semantic integrity of the output. Moreover, the evaluation process only involves weighted summation operations, making it computationally friendly to MPC environments. Therefore, the proposed composite importance evaluation method significantly reduces the generation overhead of NLG models in secure inference scenarios while better maintaining the model's generative capabilities by introducing only a small evaluation overhead.

[0066] In existing research, a commonly used metric for evaluating token importance is the average attention weight. The attention weight in the Transformer model is calculated as follows: Where A is an n×n matrix, and n represents the number of tokens. Each element a in the attention weight matrix A... i,jThis represents the correlation between the i-th and j-th tokens; a larger value indicates a higher level of attention. Therefore, the token importance assessment metric based on attention weight can be expressed as:

[0067]

[0068] Among them I t [i] represents the importance of the i-th token, and H is the number of attention heads to focus on when evaluating the importance of a token. For example, to evaluate the importance of the output token of the l-th layer, it is necessary to accumulate the weights of each attention head on token i in all previous layers. Summing the weight matrix A in the vertical direction means calculating the cumulative correlation of all n tokens on token i.

[0069] As can be seen from expression (8), relying solely on attention weights as an evaluation metric for token importance is affected by the different stages of the generation process. In the early stages of the generation process, the attention distribution is relatively sparse, making it easy to affect model performance due to the accidental pruning of key tokens, and it is difficult to measure the importance of tokens between different layers. To address these issues, a composite importance evaluation metric for tokens is proposed by utilizing the importance score of attention heads obtained offline in the global scope of the model. Its expression is as follows:

[0070]

[0071] Among them I comb [i] represents the composite importance of the i-th token. This is a normalized value for the importance of attention heads acquired offline, representing the relative importance of the attention heads actually involved in the generation task within the model. α is a balancing coefficient used to adjust the influence of offline attention head importance on the online token importance assessment, and can be dynamically adjusted according to actual needs.

[0072] Based on the composite importance evaluation index, the model M after offline attention head pruning... pruned The token vectors generated during the model generation process are pruned. First, the importance score matrix I of the attention head output offline is used... h_p The retained attention head weights are extracted and used to form the attention head importance matrix I in the online generation stage. h′ Normalization was then performed. Furthermore, to accommodate different implementations of attention head pruning in practical applications, the importance scores of all attention heads were retained; the actual attention head pruning process was completed offline. Then, for the online input x... iThe model executes the generation process and uses a composite evaluation index to evaluate the importance of tokens layer by layer. Based on the set pruning threshold θ, tokens are pruned, and pruned tokens will no longer appear in subsequent calculations. The specific token pruning algorithm is shown in Algorithm 2.

[0073]

[0074]

[0075] The online secure inference process can be implemented based on the Crypton framework, whose underlying secure inference protocol uses secret sharing. In Algorithm 2, the normalization operation for attention head importance can be completed offline before secure inference. During online inference, the calculation of token importance only involves addition and constant multiplication between secret values, which can all be completed locally by the secure computation participants. For the token pruning process, a secure comparison operation between the token importance score and the threshold θ needs to be implemented. A secure comparison protocol can be used to implement token pruning during secure inference. The main source of overhead in the pruning process is the secure comparison protocol. For the importance evaluation of a token sequence of length n at each layer, the protocol only needs to call the comparison protocol n times. The time complexity of the online token pruning protocol is approximately O(n), and it is independent of the number of attention heads and the embedding dimension. In actual tests, the time cost of a single secure comparison protocol is in the millisecond range. From the perspective of actual overhead, the overhead introduced by the pruning process is far less than the overall overhead of the secure inference process.

[0076] Furthermore, the setting of the token pruning threshold θ needs to be related to the input data. A cryptographically aware fine-tuning method is used, incorporating the threshold θ into the model's loss function. The optimal value of the threshold θ is automatically searched during model fine-tuning training. The fine-tuned model can further optimize performance based on offline attention-based pruning, becoming more adaptable to different downstream data. After fine-tuning, the obtained threshold θ value is fixed and used as the basis for token pruning in the online phase.

[0077] By pre-pruning the model offline using a gradient attribution algorithm and removing attention heads that contribute little to the general language task from various datasets, a streamlined pre-pruned model is generated. In the online phase, the streamlined model is used to perform token pruning, significantly reducing the overhead of the generated model. Based on the collaborative optimization framework, a token evaluation method based on composite importance is used to assess the importance of token representation vectors in the model. The traditional attention weight-based token evaluation method is modified using the attention head weights obtained in the offline phase, improving the global awareness of token pruning and thus achieving a better balance between model inference efficiency and generation capability.

[0078] Furthermore, based on the above method, this embodiment of the invention also provides an efficient and secure inference system for a natural language generation model based on Transformer, comprising: an offline processing module and an online inference module, wherein,

[0079] The offline processing module is used to input a general corpus dataset into the pre-trained generative model in the offline stage, calculate the importance of each attention head of the model on the general corpus dataset, and perform pruning operation on the attention heads of the model based on the importance to obtain the pruned generative model. The generative model adopts Transformer.

[0080] The online inference module is used to input the text data to be processed into the pruned generative model in the online stage. It evaluates the importance of tokens layer by layer using a composite evaluation index, prunes tokens based on token importance and using a pruning threshold, and obtains the generation sequence of the text data to be processed through forward propagation of the remaining tokens. The composite evaluation index is obtained by weighting the importance of the model attention head and the cumulative score of token attention in the offline stage.

[0081] To verify the effectiveness of this solution, the following explanation is based on experimental data:

[0082] The experiments chose the WikiText-2 dataset as the primary benchmark. This dataset contains approximately 2 million words of English text and is widely used in natural language model evaluation. All data underwent standard word segmentation and encoding preprocessing to adapt to the input requirements of the Transformer model. The WikiText-2 dataset is text data extracted from Wikipedia, specifically the titles and first paragraphs of Wikipedia articles; in text generation tasks, these datasets can be used to train models to generate coherent text.

[0083] The experiment uses the classic autoregressive language model GPT-2 based on the Transformer architecture as the base model. To test the performance of the method in this case on models of different sizes, two parameter scales are set. The specific model parameter configurations are shown in Table 1. Model_S and Model_M represent the Small and Medium scale models, respectively.

[0084] Table 1. Parameter configurations for two different scales of GPT-2 models.

[0085]

[0086] In addition, to evaluate the performance of the method under different input lengths, three input lengths of 64, 256 and 512 were set in the experiment, representing short sequence prompts, medium-length contexts and long document generation scenarios, respectively, covering the requirements of typical natural language generation tasks.

[0087] To verify the optimization effect of the proposed solution on model inference efficiency, the speedup ratio (SRR) is used as the evaluation index for model inference efficiency, i.e., the ratio of inference overhead between different methods, eliminating the influence of other factors such as the experimental environment. This is also a common approach used in existing methods. Perplexity (PPL) is used to evaluate the model performance before and after pruning. PPL is one of the most commonly used generation quality evaluation metrics for natural language models, used to measure the uncertainty of the model's prediction of the next token. The smaller the PPL, the stronger the modeling ability of the language model. Since the calculation of PPL is based on the model's probabilistic prediction, its value is highly dependent on the dataset, model structure, etc., making its value very sensitive. Therefore, based on the benchmark data given in GPT-2, a PPL variation within 30% of the benchmark model is considered an acceptable range for the impact on model generation performance.

[0088] In the offline phase, attention heads were pruned using a general public dataset; therefore, the experiment was conducted in a plaintext environment and implemented using the PyTorch deep learning framework. In the online phase, the secure inference environment was primarily built using the Crypton privacy-preserving framework, assuming that all participants in the inference process were semi-honest. The experimental hardware platform consisted of two ThinkStation P920 workstations, each equipped with two RTX 3090 GPUs, operating over a 10GbE Ethernet connection.

[0089] The practical effect of offline attention head pruning was evaluated on two models, such as Figure 3 As shown in (a) and (b) of the figure, the offline attention pruning method is effective. Without any fine-tuning, it can prune over 40% of attention heads while the model's perplexity increases by no more than 30%. Furthermore, offline fine-tuning of the model restores its performance. This offline fine-tuning does not incur additional overhead for the online inference process, which is a significant advantage of the method presented in this case.

[0090] The model performance before and after fine-tuning is as follows: Figure 4As shown in (a) and (b) above, after fine-tuning, the model performance recovered significantly. Model_S, even with 50% of attention heads pruned, maintained an acceptable increase in perplexity, while Model_M, under the same conditions, achieved 70% pruning of attention heads. Therefore, offline attention head pruning significantly reduces the model's computational scale while maintaining performance, providing a high-efficiency foundation model for further optimization of inference efficiency in the online phase. To ensure the stability of the overall generation performance, the fine-tuned model was used for the next step of online token pruning.

[0091] The experiment tested the inference efficiency and model performance under different token pruning rates. Based on the fine-tuned model performance, the offline attention pruning rates of the two models were set to 0.5 and 0.6, respectively. Then, token pruning was performed on the online generation process of the models under different input lengths. The experimental results are as follows: Figure 5 As shown in (a) and (b) in the figure.

[0092] Experimental results show that pruning tokens at a certain depth in model inference can improve inference efficiency without significantly affecting model performance. Specifically, when the token pruning rate in Model_S is 15%, the perplexity increases by an average of 2.3 for the three output lengths, an increase of only 10.4%. When the token pruning rate is increased to 30%, the increase in perplexity for the 256 and 512 lengths remains within an acceptable range. In Model_M, when the token pruning rate is 10%, the perplexity increases by an average of 21.4, an increase of only 5.7%. When the token pruning rate is increased to 30%, the perplexity for all three lengths remains within an acceptable range. Experimental results indicate that the online pruning method in this proposal improves efficiency while maintaining the quality of model generation. Comparing the token pruning results for different lengths, longer inputs have less impact on model generation capability when pruning the same proportion of tokens. This reflects that longer inputs contain more redundant tokens, and the proposed method can achieve greater token compression for long text inputs while maintaining the same model performance.

[0093] To further test the overall effectiveness of the proposed method in optimizing model inference efficiency, experiments were conducted to evaluate the efficiency improvement of the proposed solution over the baseline model under different model sizes and input lengths. For both models, the offline attention head pruning rate was set to 0.5, and the online token pruning rate was set to 0.3. The experimental results are as follows: Figure 6 As shown.

[0094] The experimental results show that the inference efficiency improvement is significant with increasing input length. When the input length is 64, the inference efficiency of the two models is improved by 1.4× and 1.71× respectively compared to the baseline model. When the input length increases to 256, the inference efficiency further improves to 1.54× and 1.92× respectively. When the input length reaches 512, the inference efficiency improvement is the largest, reaching 1.69× and 2.15× respectively, demonstrating the optimization capability of the proposed method for long sequence tasks. In addition, the experiment observed that, for the same input length, the inference efficiency improvement of Model_M is higher than that of Model_S. The main reason for this is that, in the same task, when the model size is larger, the proportion of attention operations with quadratic complexity is higher. Therefore, the joint pruning of attention heads and tokens in this proposal achieves a higher acceleration effect. The experimental results demonstrate that the proposed scheme can achieve better optimization results for secure inference models of natural language generation compared to a single token pruning strategy.

[0095] Furthermore, to further evaluate the advantages of the offline-online optimization framework in practical application deployment, a comprehensive analysis was conducted on the solution's performance in terms of memory usage, ease of deployment, and other indicators. The results are as follows: Figure 7 As shown in the figure, the experiment used five indicators closely related to practical application deployment to comprehensively evaluate the model optimization effect: inference speed (measured by speedup ratio), quality retention rate (evaluating the degree to which the accuracy of the pruned model is preserved relative to the original model), model compression rate (quantifying the reduction in model file size), memory saving (measuring the reduction in runtime memory usage), and deployment friendliness (comprehensively evaluating the ease of deployment and resource requirements of the model in a production environment). These different indicators reflect the comprehensive performance of the proposed solution in key dimensions such as efficiency, performance, resource consumption, and practical deployment feasibility.

[0096] Based on the comprehensive analysis results, compared with the baseline network, the proposed solution improves the model compression rate, memory saving, and inference speed through attention head pruning and token pruning. However, the optimization effects of the two-stage pruning method vary on different indicators. In practical applications, the participation level of the two stages can be flexibly set according to specific needs.

[0097] To verify the synergistic effect of offline attention head pruning and online token pruning in this scheme, ablation experiments were conducted to analyze the performance and efficiency of the two-stage pruning under different combinations of pruning rates. The input sequence length for both models was set to 256. In the offline stage, the attention head pruning rates for Model_S were set to 10%, 20%, and 40%, while those for Model_M were adjusted to 10%, 30%, and 50% to achieve greater pruning. In the online stage, the token pruning rates were set to 10% and 30%. The experimental results are shown in Table 2.

[0098] Table 2 Comparison of model performance under different combinations of attention points and token pruning rates

[0099]

[0100] The experimental results show that both offline attention head pruning and online token pruning are effective in optimizing model inference efficiency. However, comparing the two, token pruning plays a dominant role in efficiency optimization; increasing the token pruning rate significantly improves inference efficiency. For example, in Model_S, increasing the token pruning rate from 10% to 30% increases the efficiency improvement from 1.13× to 1.48×, while increasing the attention head pruning rate while keeping the token pruning rate unchanged results in a relatively smaller efficiency improvement. Similar results are observed in Model_M. This is mainly because embedding operations and linear operations closely related to the autoregressive process account for more than 60% of the total overhead in natural language generation models, while attention operations such as Softmax account for less than 30%. Attention pruning primarily affects attention computation, while token length affects almost all module operations. Therefore, the experiments also demonstrate that token compression is an effective way to improve secure inference efficiency for secure inference models of natural language generation. Furthermore, the experiment also observed that increasing the pruning rate of attention heads had a smaller impact on model performance compared to token pruning. This provides a feasible strategy for maintaining model performance and further improving efficiency under high pruning rates.

[0101] The above experiments demonstrate that the offline-online collaborative framework in this solution significantly improves the secure reasoning efficiency of the natural language generation model without increasing additional evaluation overhead, and also provides a wider range of strategy options for the optimization of secure reasoning in this model.

[0102] Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention.

[0103] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0104] The units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations are not considered to be beyond the scope of this invention.

[0105] Those skilled in the art will understand that all or part of the steps in the above methods can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk. Optionally, all or part of the steps in the above embodiments can also be implemented using one or more integrated circuits. Accordingly, each module / unit in the above embodiments can be implemented in hardware or as a software functional module. This invention is not limited to any particular combination of hardware and software.

[0106] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An efficient and secure reasoning method based on a Transformer-based natural language generation model, characterized in that, Include: In the offline phase, a general corpus dataset is input into the pre-trained generative model. The importance of each attention head of the model is calculated on the general corpus dataset. Based on the importance, the attention heads of the model are pruned to obtain the pruned generative model. The generative model adopts Transformer. In the online phase, the text data to be processed is input into the pruned generative model. The importance of the tokens is evaluated layer by layer using a composite evaluation index. Based on the token importance, the tokens are pruned using a pruning threshold. The generated sequence of the text data to be processed is obtained through forward propagation of the remaining tokens. The composite evaluation index is obtained by weighting the importance of the model attention head and the cumulative score of token attention in the offline phase.

2. The efficient and secure inference method for natural language generation models based on Transformer according to claim 1, characterized in that, The importance of each attention head of the model is calculated on a general corpus dataset, including: The integral gradient of each attention head in the model is calculated based on the text data in the general corpus dataset, so as to obtain the attention attribution score of the attention head based on the integral gradient; Based on the attention attribution score, the expectation of each attention head in the autoregressive process on the general corpus dataset is obtained, and the importance of each attention head on the general corpus dataset is determined based on the expectation.

3. The efficient and secure inference method for natural language generation models based on Transformer according to claim 2, characterized in that, The process of obtaining the attention attribution score based on the integral gradient is represented as follows: Among them, A h Let λ be the attention weight matrix of the attention head h, λ be the step size, F(x) be the function expression of the neural network model, and k be the path parameter in the integral gradient.

4. The efficient and secure inference method for natural language generation models based on Transformer according to claim 2, characterized in that, The process of determining the importance of each attention head on a general corpus dataset based on expectations is represented as follows: Among them, I h The attention head h represents the importance of the attention head h on the dataset D, T represents the maximum length of the generated token sequence, and E represents the importance of the attention head h on the dataset D. x～D [·] represents the expectation on dataset D. Let represent the attribution score of attention head h during the generation of the t-th token, where x is the data in dataset D.

5. The efficient and secure inference method for natural language generation models based on Transformer according to claim 1, characterized in that, Pruning operations are performed on the model's attention heads based on importance, including: Set pruning thresholds based on the pruning rate; If the importance of an attention head on a general corpus dataset is greater than the pruning threshold, then the attention head is retained and its importance score is recorded. If the importance of the attention head on the general corpus dataset is less than the pruning threshold, then the weight of the attention head is reset to zero. The pruned generative model is obtained based on the retained attention heads, and a scoring matrix is ​​composed of the importance scores of each attention head.

6. The efficient and secure inference method for natural language generation models based on Transformer according to claim 1, characterized in that, The calculation process for the composite evaluation index is expressed as follows: Among them, I comb [i] represents the composite importance of the i-th token. The value is a normalized representation of the importance of attention heads h acquired offline, where α is the balance coefficient, n represents the number of tokens, and H is the number of attention heads used to evaluate token importance. h [j,i] represents the correlation between the i-th and j-th tokens in the attention head h.

7. The efficient and secure inference method for natural language generation models based on Transformer according to claim 1, characterized in that, Secure inference is based on the Crypton framework. It encodes private data in the natural language generation task through secret sharing among multiple participants. It performs predictive inference by splitting the input data into multiple fragments and secretly sharing the fragments among different participants. It also uses a secure comparison protocol to perform pruning operations on tokens. The pruning threshold in the token pruning operation is based on the model loss function and the optimal value is obtained by model fine-tuning.

8. A high-efficiency and secure inference system based on a Transformer-based natural language generation model, characterized in that, It includes: an offline processing module and an online inference module, wherein, The offline processing module is used to input a general corpus dataset into the pre-trained generative model in the offline stage, calculate the importance of each attention head of the model on the general corpus dataset, and perform pruning operation on the attention heads of the model based on the importance to obtain the pruned generative model. The generative model adopts Transformer. The online inference module is used to input the text data to be processed into the pruned generative model in the online stage. It evaluates the importance of tokens layer by layer using a composite evaluation index, prunes tokens based on token importance and using a pruning threshold, and obtains the generation sequence of the text data to be processed through forward propagation of the remaining tokens. The composite evaluation index is obtained by weighting the importance of the model attention head and the cumulative score of token attention in the offline stage.

9. An electronic device, characterized in that, include: At least one processor, and a memory coupled to said at least one processor; The memory stores a computer program that can be executed by the at least one processor to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, enables the implementation of the method as described in any one of claims 1 to 7.

Images