Nonlinear layer dynamic calibration method for Transform security reasoning

Abstract

The invention provides a nonlinear layer dynamic calibration method for Transform safety reasoning, and relates to the technical field of safety reasoning. When safety reasoning is carried out on a Transformer model, a fixed approximation coefficient is not adopted, and whether significant deviation exists between the distribution ([mu] batch and [sigma] batch) of current input batch data and the static distribution ([mu] batch and [sigma] batch) in a training stage is monitored in real time through a distribution difference measure (DM). Once the deviation exceeds a threshold, the system triggers dynamic calibration, and the approximate expression of the nonlinear function is re-scaled and adjusted according to new batch distribution information, thereby ensuring that the approximate precision always matches the real-time data distribution.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of security reasoning, and particularly relates to a nonlinear layer dynamic calibration method for Transformer security reasoning. BACKGROUND

[0002] With the wide application of large models, data and model security risks are highlighted, and the security reasoning of Transformer is also concerned. In order to solve this problem, the existing work mostly adopts privacy protection technology (MPC) and homomorphic encryption (HE) of multi-party secure computation. However, due to the complex calculation of the Transformer model, the encrypted data will generate a large amount of overhead and communication delay in this process, thereby reducing the efficiency of the security reasoning of the Transformer. Therefore, many researchers start from the nonlinear calculation with large overhead, and propose various approximate optimization methods for improving the calculation efficiency and accuracy of the nonlinear activation function in the privacy computing framework.

[0003] In the field of security reasoning of Transformer, there are two commonly used approximation techniques, namely polynomial approximation and piecewise approximation. The existing main scheme aims to replace the GELU, softmax and other nonlinear functions with functions that are computationally efficient in the privacy protection technology framework, thereby avoiding complex and expensive secure comparison and secure division operations. The fixed coefficient approximation method is mainly used. Firstly, polynomial approximation finds a balance between efficiency and accuracy, specifies the degree of the polynomial, mostly cubic or sextic, and then uses the Taylor series and limit method for approximation. Secondly, piecewise approximation uses only one polynomial to replace the entire nonlinear function, which will produce a large error in the part deviating from the polynomial. Therefore, by observing the image of the nonlinear function, the input is segmented, and different polynomial degrees and coefficients are used for approximation in each segment. This approximation method improves the accuracy in the field of security reasoning of Transformer, but there is still a certain gap from the calculation of the plaintext.

[0004] Since the coefficients of the polynomial are obtained by using the traditional method of Taylor series or limit, there will still be some errors when reasoning in a variable data set. Therefore, it is necessary to propose an approximation method that takes into account the data set as much as possible.

[0005] A static distribution-aware nonlinear function approach is proposed in the NeurIPS 2024 conference. This approach performs distribution statistics on natural language processing datasets during the approximation stage, and obtains a candidate set of segment points accordingly. Traverse the candidate segment points, each candidate segment point divides the input range into multiple intervals, and fits a polynomial of a specified degree for each interval to minimize the overall error. According to the error performance under different segmentation, the optimal segment point configuration and corresponding segmented polynomial function are selected. Deploy this result in the approximation calculation of Transformer security reasoning.

[0006] A static distribution-aware nonlinear function approach is proposed in the NeurIPS 2024 conference; although the distribution-aware approximation is performed according to the dataset, the final deployment in the Transformer security reasoning process is still fixed coefficients. When the distribution of the training dataset and the test dataset has a bias, mainly the appearance of long-tail data, the reasoning result will be biased, which will be a fatal error in the medical and financial fields. SUMMARY

[0007] In view of the shortcomings of the prior art, the present application provides a nonlinear layer dynamic calibration method for Transformer security reasoning. Overcome the precision decline problem caused by fixed coefficients in the nonlinear function approximation of the existing Transformer security reasoning scheme, especially when facing the distribution deviation of the training set and the test set and the appearance of long-tail data, the reasoning result will occur fatal deviation defect.

[0008] The technical scheme adopted by the present application is:

[0009] A nonlinear layer dynamic calibration method for Transformer security reasoning, comprising the following steps:

[0010] Step 1: Preparation stage: extract static distribution characteristics, then confirm the candidate set and the approximation function, and finally configure the preset value;

[0011] Step 1.1: Determine the static distribution characteristics;

[0012] Statistical analysis is performed on the input activation values of the nonlinear layer in the Transformer model, and the static distribution characteristics, i.e. the static mean μ static and the static standard deviation σ static , are taken as the distribution reference;

[0013] Step 1.2: Determine the segment point candidate set of the static distribution;

[0014] According to the static distribution characteristics, the Markov chain is used to define the transition probability between potential segment points and segment points, the segment point positions are adjusted through state transition iteration, the next most suitable segment point is predicted, a series of segment points for dividing the input interval are obtained, and a segment point candidate set is established;

[0015] Step 1.3: Determine the static approximation function;

[0016] The segment point candidate set is traversed, the weighted least squares method is used for polynomial coefficient fitting for each segment point, the polynomial function coefficients are obtained, and finally the segment point with the minimum expected loss and the polynomial coefficients are taken as the final static approximation function;

[0017] Step 1.4: Configure the DM threshold;

[0018] An accuracy loss-based method is used to simulate various degrees of distribution drift on the static data set, and the distribution difference measurement threshold, i.e. the DM threshold, is obtained when the accuracy reaches the system intolerable loss;

[0019] Step 2: Online phase: real-time feature statistics are performed according to the input data, and the distribution difference measurement DM is calculated, and the DM is compared with the DM threshold, if the DM exceeds the DM threshold, dynamic calibration is performed, otherwise the static approximation function in the preparation phase is used for calculation;

[0020] Step 2.1: Data input and real-time statistical calculation;

[0021] After the Transformer model receives the input data of the current batch, the following calculations are performed:

[0022] (1) Data input: read the input data X batch ;

[0023] (2) Mean and standard deviation calculation: real-time statistical calculation is performed on the input data X batch , to obtain the batch mean μ batch and the batch variance σ batch ; μ batch =Mean(X batch ), σ batch =Std(X batch ); wherein Mean() represents the calculation of the mean value, and Std() represents the calculation of the standard deviation;

[0024] Step 2.2: Distribution difference measurement calculation;

[0025] The batch mean μ batch , the batch variance σ batch , the static mean μ static , and the static standard deviation σ static, calculate a distribution difference measure DM;

[0026] Step 2.3: distribution perception and calibration decision;

[0027] The distribution difference measure DM is compared with a DM threshold value to determine whether the distribution difference measure DM is greater than the DM threshold value; if the condition is met, it indicates that the current batch data has distribution drift, and the process jumps to step 2.4; if the condition is not met, it indicates that the current batch data distribution is stable, the system bypasses dynamic calibration, and directly uses static approximation configuration, i.e. jumps to step 2.5;

[0028] Step 2.4: dynamic calibration; the dynamic calibration includes segment point adjustment and polynomial coefficient scaling:

[0029] The segment point adjustment specifically includes translating and scaling the initial segment point according to the real-time batch parameter to obtain a new dynamic segment point;

[0030] The polynomial coefficient scaling specifically includes translating and scaling the polynomial function coefficient according to the real-time batch variance to obtain a new dynamic coefficient.

[0031] Step 2.5: approximation calculation;

[0032] Step 2.5.1: final calculation: using the dynamic segment point and dynamic coefficient after dynamic calibration in step 2.4, or the static approximation configuration when the condition is not met in step 2.3, to perform nonlinear function approximation calculation on the data;

[0033] Step 2.5.2: inference continues: return the calculation result to the Transformer model to continue the inference of the subsequent layers until the entire safe inference process is completed.

[0034] The beneficial effects produced by the above technical solutions are:

[0035] The present application provides a nonlinear layer dynamic calibration method for Transformer safe inference, which overcomes the defect of fixed approximation parameters in the existing Transformer safe inference scheme by introducing a real-time distribution perception mechanism in the online stage. The mechanism compares the variance and mean of the current input data with the static distribution characteristics by real-time calculation. The parameter adjustment decision process is realized based on the distribution difference measure. When the distribution difference measure exceeds the preset threshold, dynamic calibration is triggered, thereby ensuring the effectiveness and real-time performance of the nonlinear layer approximation coefficients, and the robustness of the model in complex data environment is greatly improved. The calibration mechanism effectively avoids unnecessary overhead while ensuring safety, greatly balancing the approximation efficiency and real-time accuracy.

[0036] Compared with the prior art, the technical scheme provided by the application realizes efficient and lightweight management of input data distribution change information by introducing a real-time distribution perception mechanism to dynamically calibrate the approximation parameters of the nonlinear layer. This mechanism avoids the complexity of pre-setting a large number of fixed segments or high-order polynomials to cope with all potential distribution drifts, which is equivalent to capturing and correcting the approximation error caused by long-tail data or domain migration in real time with minimal online calculation overhead. This dynamic and adaptive calibration ensures the high precision and data reliability of Transformer safe reasoning in high-risk (such as medical and financial) applications, thereby avoiding critical decision errors caused by fixed coefficient approximation bias. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 Preparation phase flowchart for embodiments of the application;

[0038] Figure 2 Online phase flowchart for embodiments of the application. DETAILED DESCRIPTION

[0039] The specific embodiments of the present application will be further described in detail below in conjunction with the drawings and examples.

[0040] A nonlinear layer dynamic calibration method for Transformer safe reasoning, comprising the following steps:

[0041] Step 1: Perform the preparation phase, as shown in Figure 1 : Extract the static distribution features, then confirm the candidate set and the approximation function, and finally configure the preset value;

[0042] Step 1.1: Determine the static distribution features;

[0043] Statistical analysis is performed on the input activation values of the nonlinear layer in the Transformer model, and the static distribution features, i.e., the static mean μ static and the static standard deviation σ static , are calculated as the distribution reference;

[0044] Step 1.2: Determine the segment point candidate set of the static distribution;

[0045] According to the static distribution features, the Markov chain is used to define the transition probability between potential segment points and segment points, and the segment point position is adjusted through state transition iteration to predict the next most suitable segment point, thereby obtaining a series of segment points for dividing the input interval, and thus establishing the segment point candidate set;

[0046] Step 1.3: Determine the static approximation function;

[0047] The set of segmentation point candidates is traversed, and for each segmentation point, a weighted least square method is used for polynomial coefficient fitting to obtain polynomial function coefficients, and finally the segmentation point with the minimum expected loss and the polynomial coefficients are taken as the final static approximation function;

[0048] Step 1.4: Configure the DM threshold;

[0049] An accuracy loss-based method is used to simulate various degrees of distribution drift on a static data set. In this embodiment, the existing static data set, the glue data set, is used to obtain the distribution difference measure threshold, i.e., the DM threshold, when the accuracy reaches a system-intolerable loss;

[0050] Step 2: Perform the online phase, as shown in Figure 2 : Real-time feature statistics are performed according to the input data, and the distribution difference measure DM is calculated. If the DM exceeds the DM threshold, dynamic calibration is performed, otherwise the static approximation function of the preparation phase is used for calculation;

[0051] Step 2.1: Data input and real-time statistical calculation;

[0052] After the Transformer model receives the input data of the current batch, the following calculations are performed:

[0053] (1) Data input: read the input data X batch ;

[0054] (2) Mean and standard deviation calculation: real-time statistical calculation is performed on the input data X batch to obtain the batch mean μ batch and the batch variance σ batch ; μ batch =Mean(X batch ), σ batch =Std(X batch ); where Mean() represents the calculation of the mean value, and Std() represents the calculation of the standard deviation;

[0055] Step 2.2: Distribution difference measure calculation;

[0056] The batch mean μ batch , the batch variance σ batch , the static mean μ static , and the static standard deviation σ static are used to calculate the distribution difference measure DM;

[0057] Step 2.3: Distribution perception and calibration decision;

[0058] The distribution difference measure DM is compared with the DM threshold to determine whether the distribution difference measure DM is greater than the DM threshold. If the condition is met, it indicates that there is distribution drift in the current batch of data, and the process jumps to step 2.4. If the condition is not met, it indicates that the current batch of data is stable, and the system bypasses dynamic calibration and directly adopts static approximate configuration, i.e., the process jumps to step 2.5.

[0059] Step 2.4: Perform dynamic calibration; the dynamic calibration includes segment point adjustment and polynomial coefficient scaling:

[0060] The segment point adjustment specifically involves: translating and scaling the initial segment points according to real-time batch parameters to obtain new dynamic segment points;

[0061] The polynomial coefficient scaling specifically involves shifting and scaling the polynomial function coefficients based on the real-time batch variance to obtain new dynamic coefficients.

[0062] Step 2.5: Perform approximate calculations;

[0063] Step 2.5.1: Final calculation: Using the dynamic segmentation points and dynamic coefficients after dynamic calibration in Step 2.4, or the static approximation configuration when the conditions are not met in Step 2.3, perform nonlinear function approximation calculation on the data;

[0064] Step 2.5.2: Inference continues: Return the calculation results to the Transformer model and continue the inference of subsequent layers until the entire security inference process is completed.

[0065] In this embodiment, the initial segmentation points and static polynomial coefficients are first obtained during the preparation phase based on minimizing the expected loss. During inference, the data is first read in, and its mean is calculated. and standard deviation The specific formula is as follows:

[0066] ;

[0067] ;

[0068] After calculating the mean and standard deviation to obtain the data distribution characteristics, the DM value is calculated using the following formula:

[0069] ;

[0070] After obtaining the DM value, compare it with the DM threshold. If it exceeds the DM threshold, then make dynamic adjustments. The adjustment formula for the segment points is as follows:

[0071] ;

[0072] in, denotes the new segment point, denotes the initial segment point of the preparation phase; the adjustment formula of the polynomial coefficients is as follows:

[0073] ;

[0074] wherein, denotes the value of the nth coefficient of the polynomial after adjustment, denotes the nth coefficient of the polynomial of the preparation phase.

[0075] Otherwise, the initial segment point and the polynomial coefficients of the preparation phase will be adopted. Finally, the polynomial approximation operation of the nonlinear function is performed.

[0076] The key of this mechanism is that all dynamic adjustments are based on efficient multiplication, division and addition and subtraction, avoiding the security comparison or security lookup operation with huge overhead in privacy calculation, so as to find the optimal balance point among real-time performance, security and accuracy.

[0077] Each embodiment in the present application is described in a progressive manner, and the same and similar parts between each embodiment can be referred to each other. Each embodiment mainly explains the difference from other embodiments.

[0078] The protection scope of the present application is not limited to the above-mentioned embodiments. Obviously, those skilled in the art can make various modifications and changes to the present disclosure without departing from the scope and spirit of the present disclosure. If these modifications and changes belong to the scope of the present method and its equivalent technology, the present disclosure also includes these modifications and changes.

Claims

1. A nonlinear layer dynamic calibration method for Transformer secure inference, characterized in that, The method comprises the following steps: Step 1: a preparation phase is performed: static distribution features are extracted, then a candidate set and an approximate function are confirmed, and finally preset values are configured; Step 2: an online phase is performed: real-time feature statistics are performed according to input data, a distribution difference measure DM is calculated, the DM is compared with a DM threshold value, if the DM exceeds the DM threshold value, dynamic calibration is performed, otherwise, the static approximate function in the preparation phase is used for calculation.

2. The method of claim 1, wherein, The step 1 specifically comprises the following steps: Step 1.1: static distribution features are determined; Statistical analysis is performed on the input activation values of the nonlinear layer in the Transformer model, and the static distribution characteristics, i.e., the static mean μ static and the static standard deviation σ static , are taken as the distribution reference; Step 1.2: a candidate set of segmentation points of the static distribution is determined; Step 1.3: a static approximate function is determined; The candidate set of segmentation points is traversed, and for each segmentation point, a weighted least square method is used for polynomial coefficient fitting to obtain polynomial function coefficients, and finally the segmentation point with the minimum expected loss and the polynomial coefficients are taken as the final static approximate function; Step 1.4: a DM threshold value is configured; An accuracy loss-based method is used to simulate various degrees of distribution drift on a static data set to obtain a distribution difference measure threshold value that makes the accuracy reach a system intolerable loss, that is, the DM threshold value.

3. The method of claim 1, wherein, The step 1.2 specifically comprises: according to the static distribution features, a Markov chain is used to define the transition probability between potential segment points and segment points, the segment point position is adjusted through state transition iteration, the next most suitable segment point is predicted, a series of segmentation points for dividing the input interval are obtained, and thus a candidate set of segmentation points is established.

4. The method of claim 1, wherein, The step 2 specifically comprises the following steps: Step 2.1: data input and real-time statistical calculation; Step 2.2: distribution difference measure calculation; Using the batch mean μ batch , the batch variance σ batch , and the static mean μ static , the static standard deviation σ static , a distribution difference measure DM is calculated; Step 2.3: distribution perception and calibration decision; The distribution difference measure DM is compared with the DM threshold value to determine whether the DM is greater than the DM threshold value; if the condition is met, it indicates that the current batch of data has distribution drift, and step 2.4 is jumped to; if the condition is not met, it indicates that the current batch of data is stable, the system bypasses dynamic calibration, and directly uses static approximate configuration, that is, step 2.5 is jumped to; Step 2.4: dynamic calibration; the dynamic calibration comprises segment point adjustment and polynomial coefficient scaling: Step 2.5: approximate calculation.

5. The method of claim 4, wherein, The step 2.1 specifically comprises: after the Transformer model receives the input data of the current batch, the following calculation is performed: (1) Data input: read input data X batch ; (2) Mean, standard deviation calculation: real-time statistical calculation is performed on the input data X batch to obtain the batch mean μ batch and the batch variance σ batch ; μ batch =Mean(X batch ), σ batch =Std(X batch ); wherein Mean() represents the calculation of the mean value, and Std() represents the calculation of the standard deviation.

6. The method of claim 4, wherein, The segment point adjustment in the step 2.4 specifically comprises: the initial segmentation point is translated and scaled according to the real-time batch parameters to obtain a new dynamic segmentation point; The polynomial coefficient scaling specifically comprises: the polynomial function coefficients are translated and scaled according to the real-time batch variance to obtain new dynamic coefficients.

7. The method of claim 4, wherein, The step 2.5 comprises the following steps: Step 2.5.1: final calculation: the dynamic segmentation point and the dynamic coefficient after the dynamic calibration in the step 2.4, or the static approximate configuration when the condition in the step 2.3 is not met, are used for nonlinear function approximate calculation on the data; Step 2.5.2: reasoning continues: the calculation result is returned to the Transformer model, and the reasoning of the subsequent layers continues until the entire safe reasoning process is completed.

Images