A mixed strategy game-based adversarial training fine-tuning method

Abstract

The application discloses a kind of based on hybrid strategy game's fine tuning method of adversarial training.The steps of the present application are as follows: S1: determine pre-training model, target data set, training task;S2: fine-tune pre-training model;S3: design the fine tuning objective function of adversarial training based on hybrid strategy game;S4: solve hybrid strategy game;S5: generate and update adversarial disturbance;S6: update model parameters;S7: train and evaluate model.The present application includes introducing hybrid strategy game into the fine tuning of pre-training model in adversarial training, and the game theory method-entropy mirror descending method is used to derive Nash equilibrium to solve the above game.In addition, the present application also uses sampling theorem and stochastic gradient Langevin dynamics sampling method to simplify the method into a kind of performance optimization practical algorithm.The model trained by the method of the present application can be improved in generalization performance and robust performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of machine learning, and in particular to a method and device for fine-tuning large-scale pre-trained models based on mixed strategy game-based adversarial training. BACKGROUND

[0002] Pre-trained models based on large-scale text data (such as BERT, GPT, T5, etc.) have made significant progress in almost all natural language processing tasks. For example, the basic technical breakthrough of BERT is to use a bidirectional training Transformer and attention model to perform language modeling. Compared with early studies of text sequences combined with left-to-right or bidirectional training, a bidirectional language model can better understand the language context. BERT uses an attention mechanism and a Transformer that learns the context relationship between words. The Transformer consists of two independent parts - the encoder and the decoder. The encoder reads the input text, and the decoder generates predictions for the task. Compared with traditional directional models (such as LSTM, etc.) that sequentially read input text, the encoder of the Transformer reads the entire sequence of words at once.

[0003] For different types of downstream tasks such as text classification and Q&A, fine-tuning can be achieved by training the model. Although the large number of parameters of the pre-trained model enhances its ability, it also makes the development, training and use of the model difficult. The main problem is that during the fine-tuning process, the pre-trained model can overfit the training data of the target task, resulting in poor generalization performance. Recent studies have shown that combining fine-tuning with adversarial training can successfully alleviate the above problems and improve the generalization ability of the model in downstream tasks. In addition, adversarial training in the fine-tuning stage is mainly used as a regularization method to prevent overfitting, rather than protecting the model from adversarial attacks as widely used in computer vision. SUMMARY

[0004] The technical problem solved by the present application is to provide a fine-tuning method based on mixed strategy game-based adversarial training, which can improve the overfitting problem in the fine-tuning process of the pre-trained model of natural language processing and improve the generalization ability and defense against adversarial attack ability of the model. One or more embodiments in the specification describe fine-tuning of large-scale models from the checkpoints of pre-trained models, which have been proven to be effective for various natural language processing tasks. However, due to the non-convexity of the target, the previous adversarial training method is prone to converge to a local optimum.

[0005] In view of the deficiencies of the prior art, the present application provides a fine-tuning method based on mixed strategy game-based adversarial training.

[0006] The technical solution adopted by the present invention to solve this technical problem includes the following steps:

[0007] S1: Determine the pre-trained model, target dataset, and training task;

[0008] S2: Fine-tuning of the pre-trained model;

[0009] S3: Design of the objective function for fine-tuning adversarial training based on hybrid strategy game theory;

[0010] S4: Solving mixed-strategy games;

[0011] S5: Countering disturbance generation and updates;

[0012] S6: Model parameter update;

[0013] S7: Model Training and Evaluation.

[0014] The goal of steps S1 and S2 is to obtain a pre-trained model that has been initially fine-tuned for the target dataset and task.

[0015] The specific implementation process for determining the pre-trained model, target dataset, and training task in step S1 is as follows:

[0016] 1-1. Determine the target pre-trained model f θ (·) and the target dataset D. Wherein, the target pre-trained model f θ The (·) already contains the original model parameters θ. The target dataset D = {(x, y)}, where x represents a data sample and y represents the corresponding label.

[0017] 1-2. According to f θ (·) and D, confirming the training task and the fine-tuning objective function L(f) θ (x), y). Simultaneously, modify f according to the selected target task. θ The top-level model structure (·). For example, for a multi-class classification model with class C as the target model, f can be... θ The top-level structure of (·) is replaced with a fully connected layer with an output dimension of C.

[0018] The fine-tuning of the pre-trained model described in step S2 is implemented as follows:

[0019] 2-1. Randomly sample N data points from the target dataset D and input them into model f. θ (·)

[0020] 2-2. For a given fine-tuning objective function L(f) θ (x), y), calculate the loss value, perform backpropagation, and update the model parameters θ.

[0021] 2-3. Repeat steps 2-1 to 2-2 until the model converges, obtaining the fine-tuned model f. θ (·)

[0022] The goal of steps S3-S6 is to further optimize the performance of the initially fine-tuned model through adversarial training. This invention treats adversarial training as a game and improves it using hybrid strategies from game theory. This invention argues that adversarial training is a two-way game of complete information between the model and the adversarial perturbation. Existing adversarial training conforms to pure strategy games, where both sides' strategies are specifically determined. In contrast, this invention extends adversarial training to hybrid strategy games where strategies are probabilistic, transforming existing strategies into probabilistic ones.

[0023] The specific implementation process for designing the adversarial training fine-tuning objective function based on hybrid strategy game theory in step S3 is as follows:

[0024] 3-1. Design of Objective Function Fine-tuning for Adversarial Training Based on Hybrid Policy Game Theory. In general pre-trained models, such as BERT, the model parameters θ are considered deterministic general variables. In this invention, the values ​​of the parameters can be referred to as the model's policy. Due to the continuity of parameter values, the model has an infinite number of policies. However, regardless of the optimizer used, each parameter update is deterministic, meaning the model only chooses one pure policy each time, thus remaining a pure policy. To improve the model's generalization performance during adversarial training, this invention transforms the model from a pure policy to a hybrid policy by allowing the model parameters θ to follow a probability distribution, i.e., transforming θ into a continuous random variable. Thus, during model training, the model updates its distribution rather than choosing a deterministic policy value. Similarly, this invention also transforms the adversarial perturbation δ from a deterministic perturbation value into a continuous random variable. Therefore, this invention redefines the hybrid policy game for adversarial training as follows: the two players are the model and the adversarial perturbation, their policies are the distributions of their parameters, and the payoff is the value of the objective function.

[0025] For the model f after initial fine-tuning θ (·) and the target dataset D, the adversarial training fine-tuning objective is set as follows:

[0026]

[0027] In the objective function above, Θ represents the set of probability distributions that θ follows, and Δ represents the set of probability distributions that δ follows. l(f θ (x+δ),f θ(x) is used to describe the similarity between the generated adversarial sample and the original sample with respect to the target model. For a given dataset D, a batch of data B is randomly sampled. The goal is to maximize the difference between the adversarial perturbation and the original data while minimizing the model's loss on the batch training data B. λ represents the tuning parameter between the two. Further, this invention uses M(Θ) and M(Δ) to represent the set of all Borel probability measures on Θ and Δ, transforming the original adversarial training objective function into the following Min-Max function:

[0028]

[0029] The specific implementation process of solving the hybrid strategy game in step S4 is as follows:

[0030] 4-1. The Min-Max optimization objective obtained in step S3 is a Nash equilibrium game problem. This invention uses the Entropic Mirror Descent (EMD) algorithm to solve this game problem. Let P be the above Min-Max optimization objective, then it can be expressed by the following formula:

[0031]

[0032] Where, μ t and v t Let be the perturbation and parameter distribution obtained in round t, respectively. Given a random variable z, its corresponding gradient h, and learning rate η, the MD iteration in EMD can be expressed as:

[0033]

[0034] Extending the MD iteration process to infinite dimensions yields...

[0035]

[0036] Since the density functions of μ and v are unavailable, this invention employs a common method: using empirical averages to replace the expected values ​​of the corresponding distributions.

[0037]

[0038]

[0039] in, K represents δ The mean δ of the samples. K represents θ The mean θ of the samples. Meanwhile, the MD iteration can also be expressed in a more easily manageable form:

[0040]

[0041] In summary, through the MD iteration of the T round, the present invention can solve the above-mentioned Nash equilibrium game problem.

[0042] For the generation and update of the adversarial perturbation described in step S5, according to the iterative solution process described in step S4, first calculate the empirical average of the adversarial samples. The specific implementation process is as follows:

[0043] 5-1. Given the number of times K used to calculate the empirical average of the adversarial samples in each round δ .

[0044] 5-2. Initialize the initial distribution of the adversarial samples in the t-th (t < T) round and the empirical average Input the batch training data B with a size of n, and use stochastic gradient Langevin dynamics sampling to update

[0045]

[0046] where γ t represents the sampling step size, ε is the thermal noise, and ξ = N(0,1) is the standard normal distribution.

[0047] 5-3. Calculate according to the following formula

[0048]

[0049] where β is a hyperparameter used to balance the influence of the historical mean and the current distribution on the empirical mean.

[0050] 5-4. Repeat steps 5-2 to 5-3 K δ times to obtain the empirical average of the adversarial samples in the t-th round

[0051] For the update of the model parameters described in step S6, according to the iterative solution process described in step S4, it is necessary to calculate the empirical average of the model parameters. The specific implementation process is as follows:

[0052] 6-1. Given the number of times K used to calculate the empirical average of the model parameters in each round of adversarial training θ .

[0053] 6-2. Initialize the model parameters in the t-th (t < T) round of the model and the empirical average On the batch training data B with an input data size of n, according to the empirical average of the adversarial perturbations generated in step S4 Use stochastic gradient Langevin dynamics sampling to update

[0054]

[0055] 6-3. Calculate according to the following formula

[0056]

[0057] Where β is a hyperparameter used to balance the influence of historical mean and current distribution on empirical mean.

[0058] 6-4. Repeat 6-2 to 6-3K θ Next, the empirical average of the model parameters in round t is obtained. And update the model weights:

[0059]

[0060] The specific implementation process for model training and evaluation described in step S7 is as follows:

[0061] 7-1. Repeat steps S5 to S6T times to obtain the model optimized through adversarial training.

[0062] 7-2. Evaluate model performance using test data. To verify the effectiveness of the proposed method, the model fine-tuned through adversarial training can be used. Compared to the model f that has not undergone adversarial training and fine-tuning θ Performance comparisons were made on test data.

[0063] The beneficial effects of this invention are as follows:

[0064] This invention restructures adversarial training from a game theory perspective, considering hybrid strategies and introducing a complete policy space. Methodologically, it derives the Nash equilibrium for hybrid strategy adversarial training using the entropy mirror descent method, and invents a novel hybrid strategy adversarial training algorithm. Numerically, it verifies that when applied to large-scale pre-trained models such as BERT and RoBERTa, the model trained using the method described in this application outperforms existing techniques in terms of generalization and robustness. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, some of the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Among them:

[0066] Figure 1 This is a schematic diagram of the neural network model of the present invention for a sentence input task in natural language processing classification.Figure 1 A represents the input of two sentences. Figure 1 .B represents the input of a sentence;

[0067] Figure 2 This is a flowchart illustrating the fine-tuning process of a pre-trained model;

[0068] Figure 3 This is a flowchart illustrating the process of fine-tuning adversarial training of a pre-trained model.

[0069] Figure 4 This is a flowchart illustrating the hybrid strategy adversarial training fine-tuning algorithm proposed in this invention.

[0070] Figure 5 This is the pseudocode for the training algorithm obtained using the entropy mirror descent algorithm;

[0071] Figure 6 This is the pseudocode of the training algorithm obtained using the stochastic gradient Langevin dynamics sampling method. Detailed Implementation

[0072] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of this application.

[0073] A method for adversarial training fine-tuning based on hybrid strategy game theory includes two parts: fine-tuning training on a target dataset and adversarial training fine-tuning based on hybrid game theory strategies. The specific implementation is as follows:

[0074] S1: Determine the pre-trained model, target dataset, and training task;

[0075] S2: Fine-tuning of the pre-trained model;

[0076] S3: Design of the objective function for fine-tuning adversarial training based on hybrid strategy game theory;

[0077] S4: Solving mixed-strategy games;

[0078] S5: Countering disturbance generation and updates;

[0079] S6: Model parameter update;

[0080] S7: Model Training and Evaluation.

[0081] Step S1 determines the pre-trained model, target dataset, and training task. The specific process is as follows:

[0082] 1-1. This embodiment uses the BERT model as the target pre-trained model and obtains the publicly available BERT model pre-training weights θ. This embodiment uses text classification as the target task, selects a text classification target dataset (such as AGNews), and preprocesses the data.

[0083] For the input of downstream tasks, please refer to Figure 1 As shown, the input to downstream tasks can be a single sentence or a pair of sentences. To enable BERT to handle all these tasks, its input can be either a single sentence or a pair of sentences. Specifically, a sentence can be a continuous text, not necessarily a semantically meaningful sentence. BERT's input is called a sequence, which can be a single sentence or two sentences. Single-sentence and double-sentence cases are possible, but BERT typically assigns a special notation [CLS] to the first word of the sequence, representing classification. The purpose of this notation is to ensure that BERT's final output represents information about the entire sequence, such as information at the sentence level. Figure 1 The input can be seen as composed of [CLS] and [SEP]. Each token enters the BERT model and yields its embedding representation.

[0084] 1-2. Modify the top-level model of BERT to a 3-layer fully connected network with an output dimension of 4, and use cross-entropy as the loss function for the target task.

[0085] Step S2, fine-tuning the pre-trained model, is as follows:

[0086] 2-1. On the text dataset processed in step 1-1, randomly sample batches of data with a batch size of N and input them into the modified BERT model;

[0087] 2-2. Using Adam as the optimizer, the modified BERT weights θ are fine-tuned through backpropagation;

[0088] 2-3. Repeat steps 2-1 to 2-2 until the model converges and the target model f is obtained. θ The specific process is as follows: Figure 2 As shown.

[0089] Step S3 involves designing the objective function for adversarial training based on hybrid strategy games. The specific process is as follows:

[0090] 3-1. This invention treats the model parameters and adversarial perturbations as two parties in a game and employs a hybrid strategy to solve the game process. For a batch of training data B sampled from the target dataset D, the game process can be represented by the following Min-Max objective function:

[0091]

[0092] Where μ represents the distribution that θ follows, v represents the distribution that δ follows, Θ represents the set of all possible distributions to which θ belongs, and Δ represents the set of all possible distributions to which ξ belongs. M(Θ) and M(Δ) represent the set of all Borel probability measures on Θ and Δ, respectively. L(f θ (x),y) are the target task functions determined in S1, l(f θ (x+δ),f θ (x)) is the adversarial training loss function. In the classification task of this embodiment, l(·) is chosen as the KL-divergence value, i.e. In regression tasks, l is typically the L2 distance loss, i.e., l(a,b) = (ab). 2 λ is the tuning parameter of the two loss functions.

[0093] Step S4, solving the mixed-strategy game, is as follows:

[0094] 4-1 This invention employs the Min-Max hybrid strategy game problem designed in step S3 of the Entropic Mirror Descent (EMD) algorithm. Let F be the above Min-Max optimization objective, then δ and θ are updated using the following formula:

[0095]

[0096] Where, μ t and v t Let be the perturbation and parameter distribution obtained in round t, respectively. Extending the MD iteration process to infinite dimensions and performing T adversarial training iterations, we obtain:

[0097]

[0098]

[0099] The adversarial training process proposed in this invention is as follows: Figure 4 As shown, the pseudocode for the MD iterative solution process is as follows: Figure 5 As shown.

[0100] Step S5 involves generating and updating adversarial data, and the specific process is as follows:

[0101] 5-1. Define the empirical average number of adversarial examples, K.δ = K.

[0102] 5 - 2. Initialize the initial distribution of adversarial samples for the t-th (t < T) round as an F-dimensional random vector, where F is the embedding dimension of the text in the BERT model. At the same time, initialize the empirical mean Randomly sample n text samples {x i} n from the processed target text dataset, input them into the fine-tuned BERT model, solve according to the hybrid strategy game proposed in step S4, and use stochastic gradient Langevin dynamics sampling to update

[0103]

[0104] where γ t represents the sampling step size, ε is the thermal noise, and ξ = N(0, 1) is the standard normal distribution.

[0105] 5 - 3. Perturb the text representation vector calculated in the current training round add it to its empirical mean, and update the empirical mean of the perturbation:

[0106]

[0107] where β is a hyperparameter used to balance the influence of the historical mean and the current distribution on the empirical mean.

[0108] 5 - 4. Repeat steps 5 - 2 to 5 - 3 K <​​​​​​​​​​​​​​​​​​​​​​​​​

[0113] 6-3. Calculate the BERT model parameters obtained in the current training round. Incorporate its empirical average and update the empirical average of the perturbation.

[0114]

[0115] Where β is a hyperparameter used to balance the influence of historical mean and current distribution on empirical mean.

[0116] 6-4. Repeat 6-2 to 6-3K θ Next, the empirical average of the model parameters in round t is obtained. And update the BERT model weights:

[0117]

[0118] Step S7: Model training and evaluation, the specific process is as follows:

[0119] 7-1. Repeat steps S5 to S6T times to obtain the BERT model for text classification after adversarial training optimization.

[0120] 7-2. Select test data from the chosen target text dataset and input them into the BERT model after adversarial training and fine-tuning. And the BERT model f that has not been trained and fine-tuned adversarially θ The performance of the two methods was compared to demonstrate the effectiveness of the adversarial training method based on hybrid strategy game theory proposed in this invention.

[0121] In summary, Figure 4 This is a flowchart illustrating the hybrid strategy adversarial training fine-tuning algorithm proposed in this application. Generally, through... Figure 6 The pseudocode shown can implement the entire model parameter update process of this invention.

[0122] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for fine-tuning adversarial training based on hybrid strategy games, characterized in that... Includes the following steps: S1: Determine the pre-trained model, target dataset, and training task. Take text classification as the target task, select the text classification target dataset, and preprocess the data. S2: Fine-tuning the pre-trained model; S3: Design an adversarial training fine-tuning objective function based on hybrid strategy game theory; S4: Solve mixed-strategy games; S5: Generate and update against perturbations; S6: Update model parameters; S7: Training and evaluating the model; Step S3 involves designing an adversarial training fine-tuning objective function based on hybrid strategy game theory, specifically implemented as follows: The model is transformed from a pure policy to a mixed policy, allowing the model parameters θ to follow a probability distribution, i.e., θ is transformed into a continuous random variable; the adversarial perturbation δ is transformed from a deterministic perturbation value into a continuous random variable that follows a probability distribution; the mixed policy game of adversarial training is redefined as follows: the two players are the model and the adversarial perturbation, the strategies of the two players are the distributions of their parameters, and the payoff of the game is the value of the objective function. Given a target pre-trained model f θ (·), the downstream target dataset D, and the adversarial training objective are represented by the following game process: Among them, l(f) θ (x+δ),f θ Let (x) denote the adversarial training objective function, and λ be the tuning parameter; consider the set of all probability distributions on Θ and Δ; if M(Θ) and M(Δ) denote the set of all Borel probability measures on Θ and Δ, then the original adversarial training objective function is transformed into the following Min-Max function: The specific method for step S4 is as follows: The Min-Max game optimization is solved using the entropy mirror descent algorithm: Where P represents the optimization objective of the aforementioned Min-Max game; μ t and v t Let be the perturbation and parameter distribution obtained in round t, respectively; given a random variable z, corresponding gradient h and learning rate η, the MD iteration in EMD can be expressed as: Extending the MD iteration process to infinite dimensions yields: Since the density functions of μ and v are unavailable, empirical averages are used instead of the expected values ​​of the corresponding distributions. in, K represents δ The mean δ of the samples. K represents θ The mean θ of the samples; mean MD iteration is also expressed in a more manageable form: Therefore, the above Nash equilibrium game problem can be solved by T rounds of MD iteration.

2. The adversarial training fine-tuning method based on hybrid strategy game theory according to claim 1, characterized in that... Step S1 determines the pre-trained model, target dataset, and training task, and is implemented as follows: 1-1. Determine the target pre-trained model f θ (·) and the target dataset D, where the target pre-trained model f θ (·) already contains the original model parameters θ; the target dataset D = {(x,y)}, where x represents the data sample and y represents the corresponding label; 1-2. Based on the target pre-trained model f θ (·) and the target dataset D, confirming the training task and the fine-tuning objective function L(f) θ (x),y); 1-3. Improve the target pre-trained model f according to the selected target task. θ The top-level structure of (·).

3. The adversarial training fine-tuning method based on hybrid strategy game theory according to claim 2, characterized in that... The fine-tuning of the pre-trained model described in step S2 is specifically implemented as follows: 2-1. Randomly sample N data points from the target dataset D and input them into model f. θ (·); 2-2. For a given fine-tuning objective function L(f) θ (x), y), calculate the loss value, perform backpropagation, and update the model parameters θ; 2-3. Repeat steps 2-1 to 2-2 until the model converges, obtaining the fine-tuned model f. θ (·) 4. The adversarial training fine-tuning method based on hybrid strategy game theory according to claim 3, characterized in that... The specific method for step S5 is as follows: 5-1. Given the number of times K is used to calculate the empirical average of adversarial examples in each round. δ ; 5-2. Initialize the initial distribution of adversarial examples in round t and experience average Given a batch of training data B of size n, we use stochastic gradient Langevin dynamics sampling to obtain an iterative formula for perturbation updates, and then update... Where, 0≤k <K δ -1, Let γ represent the adversarial perturbation distribution of the k-th sample during the t-th round of adversarial training optimization. t The sampling step size is represented by ε, where ε is thermal noise and ξ = N(0,1) is a standard normal distribution. This indicates calculating the gradient of l with respect to δ; 5-3. Calculate according to the following formula Wherein, β is a hyperparameter used to balance the influence of historical mean and current distribution on empirical mean; 5-4. Repeat steps 5-2 to 5-3K. δ Next, the empirical average of the adversarial examples in round t is obtained.

5. The adversarial training fine-tuning method based on hybrid strategy game theory according to claim 4, characterized in that... The specific method for step S6 is as follows: 6-1. Given K, the number of times the empirical average of the model parameters is calculated in each round of adversarial training. θ ; 6-2. Initialize the model parameters in round t and experience average On a batch training data B with an input data size of n, the empirical average of the adversarial perturbation generated in step S4 is applied. Update using stochastic gradient Langevin dynamics sampling 6-3. Calculate according to the following formula Where, 0≤k <K θ -1, β are hyperparameters used to balance the influence of historical mean and current distribution on empirical mean; 6-4. Repeat steps 6-2 to 6-3. θ Second-rate, This represents the distribution of model parameters sampled in the k-th iteration during the t-th round of adversarial training optimization, ultimately yielding the empirical average of the model parameters in the t-th round. And update the model weights:

Images