MI-FGSM anti-attack method based on reinforcement learning dynamic optimization

Abstract

The invention relates to an MI-FGSM attack resisting method based on reinforcement learning dynamic optimization, belongs to deep learning security, and solves the problems of fixed parameters, low attack success rate and insufficient concealment in the existing attack resisting process. Updating parameters of the actor network and the commentator network in the strategy network through iterative training to obtain a trained strategy network; converting a to-be-attacked image into an initial state vector, inputting the initial state vector into the trained strategy network, outputting a momentum coefficient and an iteration step size through an actor network, and evaluating a state value of a current state vector through a commentator network; and if the state value is lower than a preset threshold value, random noise is injected into the current state vector to form a disturbed state vector, the disturbed state vector is input into the trained strategy network again until the state value reaches or exceeds the preset threshold value, and the generative adversarial sample is executed based on the current momentum coefficient and the iteration step length.

Description

Technical Field

[0001] This invention relates to the field of deep learning security, and in particular to a method for adversarial attacks using MI-FGSM based on reinforcement learning dynamic optimization. Background Technology

[0002] Adversarial attacks, a key technology in deep learning security, aim to induce incorrect predictions in target models by applying imperceptible perturbations. MI-FGSM, as a mainstream iterative attack method, demonstrates a high success rate in various attack scenarios by introducing a momentum term to stabilize the gradient update direction.

[0003] However, traditional MI-FGSM methods rely on manually preset fixed parameters (such as momentum coefficients and iteration step sizes). These parameters lack adaptability under different model structures and data distributions, resulting in limited attack efficiency and susceptibility to model defense mechanisms. Although recent research has attempted to introduce reinforcement learning for parameter optimization, existing methods still have significant shortcomings in state space design, policy network architecture, and training stability, making it difficult to achieve accurate decision-making and efficient attacks in dynamic environments. Fixed-parameter policies cannot adaptively adjust according to the attack state, leading to large fluctuations in attack success rates; the lack of an effective state evaluation mechanism during iteration results in low attack success rates; and traditional methods have insufficient control over the magnitude of perturbations and attack step sizes, making it difficult to balance attack efficiency and stealth. Summary of the Invention

[0004] Based on the above analysis, the embodiments of the present invention aim to provide a MI-FGSM adversarial attack method based on reinforcement learning dynamic optimization, in order to solve the problems of fixed parameters, low attack success rate and insufficient concealment in the existing adversarial attack process.

[0005] On one hand, embodiments of the present invention provide a method for MI-FGSM adversarial attacks based on reinforcement learning dynamic optimization, characterized in that it includes:

[0006] A policy network is constructed based on an actor-critic architecture. The parameters of the actor network and the critic network in the policy network are updated through iterative training to obtain the trained policy network.

[0007] The image to be attacked is converted into an initial state vector. The initial state vector is input into the trained policy network. The momentum coefficient and iteration step size are output through the actor network. The state value of the current state vector is evaluated through the critic network.

[0008] If the state value is lower than a preset threshold, random noise is injected into the current state vector to form a perturbed state vector, and the perturbed state vector is re-inputted into the trained policy network until the state value reaches or exceeds the preset threshold. Based on the current momentum coefficient and iteration step size, adversarial examples are generated.

[0009] Furthermore, the process of converting the image to be attacked into an initial state vector includes:

[0010] The image is converted into a tensor format that the target model can process, resulting in an image sample tensor.

[0011] Initialize the current cumulative perturbation tensor to zero;

[0012] Initialize the normalized iteration step count to zero;

[0013] Calculate the initial normalized perturbation norm based on the image sample tensor;

[0014] The image to be attacked is input into the target model, the prediction result is obtained and converted into a one-hot vector, and the image prediction one-hot vector is obtained.

[0015] The predicted one-hot vector is also used as an adversarial example to predict the one-hot vector;

[0016] The image tensor, the current cumulative perturbation tensor, the normalized iteration step number, the normalized perturbation norm, the image prediction one-hot vector, and the adversarial example prediction one-hot vector are concatenated to form the initial state vector.

[0017] Furthermore, the initial state vector is represented as:

[0018] s t =[img t ,pert t ,step t norm t ,orig_pred t curr_pred t ]

[0019] Among them, img t For image tensors; pert t This represents the current cumulative perturbation tensor; step t Normalized iteration number; norm t For the normalized perturbation norm; orig_pred t Predict one-hot vectors for an image; curr_pred t To predict one-hot vectors for adversarial examples.

[0020] Furthermore, injecting random noise into the current state vector to form the perturbed state vector includes:

[0021] Random noise with the same dimension as the current accumulated perturbation tensor is sampled from a Gaussian distribution with zero mean and adjustable variance;

[0022] A random noise matrix is ​​generated based on the sampled random noise and added to the current cumulative perturbation tensor to obtain the cumulative perturbation tensor after injecting noise;

[0023] The normalized perturbation norm after injecting noise is obtained by recalculating the cumulative perturbation tensor after injecting noise.

[0024] The cumulative perturbation tensor after injecting noise and the normalized perturbation norm are updated to the current state vector to obtain the perturbed state vector.

[0025] Furthermore, iteratively training and updating the parameters of the actor network and the critic network in the policy network includes:

[0026] The training samples are used as raw adversarial samples and input into the target model to obtain the prediction results of the training samples;

[0027] An initial state vector is generated based on the training samples and their prediction results, and this initial state vector is used as the current state vector; the following steps are executed iteratively:

[0028] Step a: Generate action vectors based on the current state vector using the policy network. The action vectors include momentum coefficients and iteration step size.

[0029] Step b: Execute the MI-FGSM attack update adversarial sample according to the action vector, obtain the next generation state vector according to the updated adversarial sample, and calculate the reward value of this iteration;

[0030] Repeat step ab using the next-generation state vector as the current state vector until the termination condition is met.

[0031] Collect the current state vector, reward value, action vector, and next-generation state vector for each iteration as transition data for each iteration;

[0032] The advantage function estimate is calculated for each iteration based on all transition data, and the parameters of the actor network and the critic network are updated based on the advantage function estimate using a proximal policy optimization algorithm, wherein each iteration includes a preset number of iterations.

[0033] Furthermore, the estimated value of the advantage function for each iteration, calculated based on the transition data of each iteration, is shown in the following formula;

[0034]

[0035] Where, δ tLet γ be the perturbation in the t-th iteration, λ be the discount factor, λ be the dominance estimation parameter, T be the maximum number of iterations, and δ be the perturbation in the t-th iteration. t+1 For the perturbation in the (t+1)th iteration, δ T-1 The perturbation in the (t-1)th iteration

[0036] δ t =r t +γV(s t+1 )-V(s t )

[0037] Where, r t Let s be the reward value for the t-th iteration. t Let s be the current state vector in the t-th iteration. t+1 Let V(s) be the next state vector for the t-th iteration. t V(s) represents the value estimate of the current state vector of the critic network for the t-th iteration. t+1 ) represents the value estimate of the next state vector of the commentator network for the t-th iteration.

[0038] Furthermore, updating the parameters of the actor network and the critic network based on the advantage function estimate using a proximal policy optimization algorithm includes:

[0039] Based on the state vectors and action vectors in the collected transition data, calculate the probability ratio of the current policy to the old policy used when collecting the transition data, in generating actions under the same state-action pair.

[0040] Based on the probability ratio and the advantage function estimate, a pruning objective function is constructed, and the parameters of the actor network are updated by maximizing the pruning objective function;

[0041] Based on the state vectors and reward values ​​in the collected transition data, a difference objective function is constructed by calculating the difference between the state value output by the critic network and the target value. The parameters of the critic network are updated by minimizing the difference objective function.

[0042] Furthermore, the reward value is shown in the following formula;

[0043]

[0044] Among them, R success R is the reward value for a successful attack. fail The penalty value for a failed attack is I, an indicator function that takes the value 1 if the condition is met, and 0 otherwise; f is the prediction function of the target model, x is the adversarial example, and R is the value of the target model. step Let δ be the iteration efficiency reward coefficient, y be the true label, and δ be the tactical efficiency reward coefficient. t Let ω be the perturbation in the t-th iteration, and let ω be the perturbation penalty coefficient.

[0045] Furthermore, the termination condition is:

[0046] The target model predicts an error or the preset maximum number of iterations has been reached.

[0047] On the other hand, embodiments of the present invention provide a MI-FGSM adversarial attack system based on reinforcement learning dynamic optimization, comprising:

[0048] The training module is used to construct a policy network based on the actor-critic architecture. The parameters of the actor network and the critic network in the policy network are updated through iterative training to obtain the trained policy network.

[0049] The output module is used to convert the image sample to be attacked into an initial state vector, input the initial state vector into the trained policy network, output the momentum coefficient and iteration step size through the actor network, and evaluate the state value of the current state vector through the critic network.

[0050] The judgment generation module is used to generate adversarial examples by executing an attack process based on the current momentum coefficient and iteration step size when the state value reaches or exceeds a preset threshold; and to inject random noise into the current state vector to form a perturbed state vector when the state value is lower than the preset threshold, and to re-input the perturbed state vector into the trained policy network until the state value reaches or exceeds the preset threshold, and then execute an attack process based on the current momentum coefficient and iteration step size to generate adversarial examples.

[0051] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0052] 1. By constructing a policy network based on an actor-critic architecture, real-time adjustment of momentum coefficients and iteration step size is achieved. The state vector, which contains multi-dimensional information such as image tensor, cumulative perturbation, and iteration step number, is taken as input. The actor network outputs the optimal action parameters, while the critic network accurately evaluates the value of the current state. This solves the limitation of traditional MI-FGSM relying on empirical fixed parameter combinations, enabling the attack process to automatically adjust parameter configuration according to different target model characteristics, input features, and attack progress stages, thereby improving the adaptability of the attack method in diverse scenarios.

[0053] 2. The system uses a commentator network to evaluate the state vector in real time. When the state value evaluated by the commentator network is lower than a set threshold, the system samples random noise from a Gaussian distribution and injects it into the current accumulated perturbation to form a perturbed state vector. This perturbed state vector is then re-input into the policy network for decision-making, ensuring a high success rate against complex defense mechanisms.

[0054] 3. The proximal strategy optimization algorithm is used to train the actor network and the critic network in a collaborative manner. The value of the action is evaluated by generalized advantage function estimation. Objective functions including pruning mechanism and value difference-based objective function are constructed to optimize the parameters of the two networks respectively. The perturbation size and attack step size are balanced to generate adversarial samples with low visual impact while maintaining a high attack success rate, thus improving the stealth while ensuring the attack effect.

[0055] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0056] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0057] Figure 1 This invention provides an embodiment of a MI-FGSM adversarial attack method based on reinforcement learning dynamic optimization. Detailed Implementation

[0058] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0059] Example 1

[0060] A specific embodiment of the present invention discloses a MI-FGSM adversarial attack method based on reinforcement learning dynamic optimization, such as... Figure 1 As shown.

[0061] S1. Construct a policy network based on the actor-critic architecture, and update the parameters of the actor network and the critic network in the policy network through iterative training to obtain the trained policy network;

[0062] The policy network employs an actor-critic architecture, comprising an actor network and a critic network. The actor network takes a state vector as input and outputs an action vector, including momentum coefficients and iteration step size. The critic network takes the current state vector as input and outputs the state value.

[0063] Specifically, iteratively training and updating the parameters of the actor network and the critic network in the policy network includes:

[0064] The training samples are used as raw adversarial samples and input into the target model to obtain the prediction results of the training samples;

[0065] An initial state vector is generated based on the training samples and their prediction results, and this initial state vector is used as the current state vector; the following steps are executed iteratively:

[0066] Step a: Generate action vectors based on the current state vector using the policy network. The action vectors include momentum coefficients and iteration step size; the action vector is represented as a. t =[μ t ,α t ], where μ t Let α be the momentum coefficient. t This is the iteration step size.

[0067] Step b: Execute the MI-FGSM attack update adversarial sample according to the action vector, obtain the next generation state vector according to the updated adversarial sample, and calculate the reward value of this iteration;

[0068] A MI-FGSM attack process is executed based on the momentum coefficient and iteration step size in the action vector. An updated adversarial example is generated based on the current adversarial example, the loss gradient of the target model, and the parameters provided by the policy network.

[0069] The next-generation state vector obtained from the updated adversarial examples includes:

[0070] Step 1: Calculate the loss gradient of the current adversarial example. The loss gradient is used to measure the difference between the model prediction and the true label, as shown in the following formula;

[0071] Where x is the current adversarial sample, δ t Let y be the perturbation in the t-th iteration, y be the true label, and J be the loss function of the target model.

[0072] Step 2: Update the cumulative gradient based on the momentum coefficient of the current iteration, as shown in the following formula;

[0073] g t+1 =μ t ·g t +g grad / ‖g grad ||1

[0074] Among them, g t+1 For the updated cumulative gradient, g t For the current accumulated gradient, μ t Let g be the momentum coefficient for the current iteration. grad Let g be the loss gradient of the current adversarial example. grad ||1 represents the L1 norm of the loss gradient of the current adversarial example.

[0075] Step 3: Update the perturbation based on the current iteration step size and the updated cumulative gradient, as shown in the following formula;

[0076] δ t+1 =δ t +α t ·sign(g t+1 )

[0077] Where, δ t+1 For the perturbation in the (t+1)th iteration, α t For the current iteration compensation, δ t Let g be the perturbation in the t-th iteration, and sign be the sign function. t+1 When g is greater than 0, the output value is +1; when g is greater than 0, the output value is +1. t+1 When g is less than 0, the output value is -1; when g is less than 0, the output value is -1. t+1 When the value is 0, the output value is 0.

[0078] Step 4: Prune the amplitude of the updated perturbation;

[0079] To ensure the stealth of adversarial examples, it is necessary to ensure that the pixel value of the total perturbation does not exceed the preset small total perturbation upper limit ε, as shown in the following formula;

[0080] δ t+1 =clip(δt) +1 ,-ε,ε)

[0081] By pruning the amplitude, the perturbation in the (t+1)th iteration is restricted to the interval [-ε, ε] to ensure the concealment of the adversarial sample.

[0082] Step 5: Add the pruned perturbation to the current adversarial sample to obtain the updated adversarial sample, as shown in the following formula;

[0083] x′=x+δ t+1 , where x′ is the updated adversarial example.

[0084] Step 6: Crop the pixel values ​​of the updated adversarial sample;

[0085] x” = clip(x′, x min ,x max ), where x min and x max Let x be the minimum and maximum pixel values ​​of the image, and x" be the newly generated adversarial example after cropping. The pixel values ​​of the newly generated adversarial example are cropped, and the pixel values ​​of the updated adversarial example are restricted to [x]. min ,x max Within [0,255]).

[0086] Step 7: Use the target model to predict the adversarial examples after pixel value cropping to obtain the updated prediction results; p t+1 =f(x”), p t+1 The updated prediction result is given by f, where f is the prediction function of the target model.

[0087] Step 8: Generate the next state vector based on the updated perturbation, the updated prediction result, and the updated iteration step number, as shown in the following formula.

[0088]

[0089] onehot(p0) is the onehot encoding of the prediction result of the original image, onehot(p t+1 ) represents the one-hot encoding of the prediction result of the new adversarial example, s t+1 This is the next state vector.

[0090] The reward value is used to evaluate the quality of the action. Generally, if the target model is misclassified in this iteration, a positive reward is obtained; otherwise, a negative reward is obtained.

[0091] The reward value is shown in the following formula;

[0092]

[0093] Among them, R success R is the reward value for a successful attack. fail The penalty value for a failed attack is I, an indicator function that takes the value 1 if the condition is met, and 0 otherwise; f is the prediction function of the target model, x is the adversarial example, and R is the value of the target model. step Let δ be the iteration efficiency reward coefficient, y be the true label, and δ be the tactical efficiency reward coefficient. t Let ω be the perturbation in the t-th iteration, and let ω be the perturbation penalty coefficient.

[0094] Repeat step ab using the next-generation state vector as the current state vector until the termination condition is met.

[0095] The next-generation state vector is used as the current state vector, and step ab is repeated until the termination condition is met. The termination condition is either that the target model's prediction result is incorrect or the preset maximum number of iterations is reached. An incorrect prediction result indicates that the predicted label output by the model is inconsistent with the true label of the original adversarial sample when the adversarial sample is input into it. In this case, the attack on the target model is successful, and the training requirements have been met; further training is unnecessary. The preset number of iterations can be set to, for example, 20-30 times. Training stops when the preset number of iterations is reached to avoid wasting resources.

[0096] The termination condition is shown in the following formula;

[0097] done=I(f(x+δ t )≠y)orI(t≥T)

[0098] Where I(·) is an indicator function, which takes the value 1 when the condition is met and 0 otherwise, f(x+δ) t ) represents the prediction result of the target model at the t-th iteration, where t is the current iteration number, T is the maximum number of iterations, x is the original adversarial sample, and y is the true label.

[0099] The current state vector, reward value, action vector, and next-generation state vector of each iteration are collected as transition data for each iteration; the current state vector, reward value, action vector, and next-generation state vector form a quadruple as transition data. Based on all transition data, the advantage function estimate for each iteration is calculated, and the parameters of the actor network and critic network are updated using a proximal policy optimization algorithm based on the advantage function estimate, wherein each iteration includes a predetermined number of iterations.

[0100] The estimated value of the advantage function for each iteration is calculated based on all transfer data, as shown in the following formula;

[0101]

[0102] Where, δ t Let γ be the perturbation in the t-th iteration, λ be the discount factor, λ be the dominance estimation parameter, T be the maximum number of iterations, and δ be the perturbation in the t-th iteration. t+1 For the perturbation in the (t+1)th iteration, δ T-1 The perturbation in the (t-1)th iteration

[0103] δ t =r t +γV(s t+1 )-V(s t )

[0104] Where, r t Let s be the reward value for the t-th iteration. t Let s be the current state vector in the t-th iteration. t+1 Let V(s) be the next state vector for the t-th iteration. t V(s) represents the value estimate of the current state vector of the critic network for the t-th iteration. t+1 ) represents the value estimate of the next state vector of the commentator network for the t-th iteration.

[0105] Furthermore, updating the parameters of the actor network and the critic network based on the advantage function estimate using a proximal policy optimization algorithm includes:

[0106] Based on the state vectors and action vectors in the collected transition data, calculate the ratio of the probability of the current policy generating an action under the same state-action pair to the probability of the old policy used when collecting the transition data; for the same state vector s t The inputs are respectively fed into the actor network to be optimized (parameter θ, representing the current strategy) and the actor network used during data collection (parameter θ). old In the old strategy, the same action vector a is generated by both strategies. t The probability ratio is shown in the following formula;

[0107]

[0108] Where, ρ t (θ) is the probability ratio, π θ (a t |s t ) represents the probability of generating an action under the current policy. The probability of generating actions using the old strategy used when collecting transfer data.

[0109] Based on the probability ratio and the advantage function estimate, a pruning objective function is constructed, and the parameters of the actor network are updated by maximizing the pruning objective function; the pruning objective function is:

[0110]

[0111] Among them, E t [] is used to calculate the expected value, ρ t (θ) represents the probability ratio, θ is the network policy parameter to be optimized, ε is the pruning parameter, and clip() is the pruning function. When the estimated advantage function value for a certain action is positive, it indicates that the action is better than the average level, and the probability of the action being selected should be increased; conversely, when the estimated advantage function value is negative, its probability should be decreased. Directly maximizing the product of the probability ratio and the advantage function will lead to an excessively large policy update step, causing training instability. Therefore, the probability ratio is restricted to the interval [1-ε, 1+ε] by the pruning function to avoid excessive changes in the policy in a single update. The pruning objective function ultimately takes the smaller value between the original ratio and the pruned ratio to ensure the stability of the policy update.

[0112] Based on the state vectors and reward values ​​in the collected transition data, a difference objective function is constructed by calculating the difference between the state value output by the critic network and the target value. The parameters of the critic network are updated by minimizing the difference objective function.

[0113] The difference objective function is expressed as:

[0114]

[0115] Among them, Et [] is used to calculate the expected value. V represents the state value output by the commentator network. targ For target value, The parameters of the critic network to be optimized.

[0116] S2. Convert the image to be attacked into an initial state vector, input the initial state vector into the trained policy network, output the momentum coefficient and iteration step size through the actor network, and evaluate the state value of the current state vector through the critic network.

[0117] The process of converting the image to be attacked into an initial state vector includes:

[0118] S21. Convert the image into a tensor format that the target model can process to obtain an image sample tensor. Image processing libraries such as Python's PIL or OpenCV can be used to read the image file and scale the pixel values ​​from 0-255 to 0-1 or -1 to 1. Finally, convert it into a multidimensional array that meets the format requirements of the target model, i.e., a tensor.

[0119] S22. Initialize the current cumulative perturbation tensor to zero; at the start of the attack, no perturbation has been added yet, so the tensor with all element values ​​of 0 is used as the current cumulative perturbation tensor, representing the total perturbation added to the image so far.

[0120] S23. Initialize the normalized iteration step count to zero; the normalized iteration step count represents the ratio of the current iteration count to the maximum allowed iteration count. In the initial state vector, since no iteration has started, the normalized iteration step count is set to zero.

[0121] S24. Calculate the initial normalized perturbation norm based on the image sample tensor; the normalized perturbation norm is a quantized representation of the current cumulative perturbation magnitude, obtained by calculating and normalizing the L2 norm or L∞ norm of the perturbation tensor. Initially, since the perturbation is zero, the normalized perturbation norm value is 0.

[0122] S25. Input the image to be attacked into the target model, obtain the prediction result and convert it into a one-hot vector to obtain the image prediction one-hot vector; the image prediction one-hot vector is obtained by inputting the original image into a pre-trained reinforcement learning model and encoding it in one-hot form, with the position corresponding to the true class being 1 and the other positions being 0.

[0123] S26. The predicted one-hot vector is simultaneously used as the adversarial example predicted one-hot vector. The adversarial example predicted one-hot vector is consistent with the original image predicted vector in the initial state vector because no perturbation has been added at this time, and the adversarial example is the original image itself.

[0124] S27. The image tensor, the current cumulative perturbation tensor, the normalized iteration step number, the normalized perturbation norm, the image prediction one-hot vector, and the adversarial example prediction one-hot vector are concatenated to form an initial state vector.

[0125] The initial state vector is represented as follows:

[0126] s t =[img t ,pert t ,step t norm t ,orig_pred t curr_pred t ]

[0127] Among them, img t For image tensors; pert t This represents the current cumulative perturbation tensor; step t Normalized iteration number; norm t For the normalized perturbation norm; orig_pred t Predict one-hot vectors for an image; curr_pred t To predict one-hot vectors for adversarial examples.

[0128] S3. If the state value is lower than the preset threshold, inject random noise into the current state vector to form a perturbed state vector, and re-input the perturbed state vector into the trained policy network until the state value reaches or exceeds the preset threshold. Generate adversarial examples based on the current momentum coefficient and iteration step size.

[0129] The state value output by the commentator network is compared with a preset threshold. For example, the preset threshold is set to 0.5. If the state value is lower than the preset threshold, random noise is injected into the current state vector, and the perturbed state vector is re-input into the trained policy network until the state value reaches or exceeds the preset threshold.

[0130] Injecting random noise into the current state vector to form a perturbed state vector includes:

[0131] S31. Sample random noise from a Gaussian distribution with the same dimension as the current cumulative perturbation tensor. The mean of the Gaussian distribution is zero and the variance is adjustable. The random noise is generated by calling a random number generator in the deep learning framework, such as using the torch.randn_like() or tf.random.normal() function to generate a Gaussian distribution noise matrix with the same dimension as the current cumulative perturbation tensor. The mean parameter is fixed to zero, and the variance is set to between 0.01 and 0.1 to control the intensity range of the noise.

[0132] S32. Generate a random noise matrix based on the sampled random noise and add it to the current cumulative perturbation tensor to obtain the cumulative perturbation tensor after injecting noise; add the noise tensor to the current cumulative perturbation tensor element by element, that is, introduce a random offset on the basis of the original attack.

[0133] S33. Based on the cumulative perturbation tensor after noise injection, recalculate the normalized perturbation norm to obtain the normalized perturbation norm after noise injection; specifically, by calculating the infinite norm of the cumulative perturbation tensor after noise injection, that is, finding the maximum value of the absolute value of all elements in the cumulative perturbation tensor after noise injection, and then dividing this maximum value by the preset perturbation upper limit value, the normalized perturbation norm after noise injection is finally obtained.

[0134] S34. Update the current state vector with the cumulative perturbation tensor after injecting noise and the normalized perturbation norm to obtain the perturbed state vector. Replace the corresponding parts in the original state vector with the newly calculated cumulative perturbation tensor and normalized perturbation norm after injecting noise, while keeping the image tensor, iteration steps, and prediction vector unchanged.

[0135] The embodiments of the present invention have at least the following beneficial effects:

[0136] 1. By constructing a policy network based on an actor-critic architecture, real-time adjustment of momentum coefficients and iteration step size is achieved. The state vector, which contains multi-dimensional information such as image tensor, cumulative perturbation, and iteration step number, is taken as input. The actor network outputs the optimal action parameters, while the critic network accurately evaluates the value of the current state. This solves the limitation of traditional MI-FGSM relying on empirical fixed parameter combinations, enabling the attack process to automatically adjust parameter configuration according to different target model characteristics, input features, and attack progress stages, thereby improving the adaptability of the attack method in diverse scenarios.

[0137] 2. The system uses a commentator network to evaluate the state vector in real time. When the state value evaluated by the commentator network is lower than a set threshold, the system samples random noise from a Gaussian distribution and injects it into the current accumulated perturbation to form a perturbed state vector. This perturbed state vector is then re-input into the policy network for decision-making, ensuring a high success rate against complex defense mechanisms.

[0138] 3. The proximal strategy optimization algorithm is used to train the actor network and the critic network in a collaborative manner. The value of the action is evaluated by generalized advantage function estimation. Objective functions including pruning mechanism and value difference-based objective function are constructed to optimize the parameters of the two networks respectively. The perturbation size and attack step size are balanced to generate adversarial samples with low visual impact while maintaining a high attack success rate, thus improving the stealth while ensuring the attack effect.

[0139] Example 2

[0140] One embodiment of a MI-FGSM adversarial attack system based on reinforcement learning dynamic optimization includes:

[0141] The training module is used to construct a policy network based on the actor-critic architecture. The parameters of the actor network and the critic network in the policy network are updated through iterative training to obtain the trained policy network.

[0142] The output module is used to convert the image sample to be attacked into an initial state vector, input the initial state vector into the trained policy network, output the momentum coefficient and iteration step size through the actor network, and evaluate the state value of the current state vector through the critic network.

[0143] The judgment generation module is used to generate adversarial examples by executing an attack process based on the current momentum coefficient and iteration step size when the state value reaches or exceeds a preset threshold; and to inject random noise into the current state vector to form a perturbed state vector when the state value is lower than the preset threshold, and to re-input the perturbed state vector into the trained policy network until the state value reaches or exceeds the preset threshold, and then execute an attack process based on the current momentum coefficient and iteration step size to generate adversarial examples.

[0144] It is understandable that the modules described in this reinforcement learning-based dynamically optimized MI-FGSM adversarial attack system are similar to those in the reference system. Figure 1 The steps described correspond to those in the reinforcement learning-based dynamic optimization-based MI-FGSM adversarial attack method. Therefore, the operations, features, and beneficial effects described above for the reinforcement learning-based dynamic optimization-based MI-FGSM adversarial attack method are also applicable to the reinforcement learning-based dynamic optimization-based MI-FGSM adversarial attack system and its constituent modules, and will not be repeated here.

[0145] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0146] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for MI-FGSM adversarial attacks based on reinforcement learning dynamic optimization, characterized in that, include: A policy network is constructed based on an actor-critic architecture. The parameters of the actor network and the critic network in the policy network are updated through iterative training to obtain the trained policy network. The image to be attacked is converted into an initial state vector. The initial state vector is input into the trained policy network. The momentum coefficient and iteration step size are output through the actor network. The state value of the current state vector is evaluated through the critic network. If the state value is lower than a preset threshold, random noise is injected into the current state vector to form a perturbed state vector, and the perturbed state vector is re-inputted into the trained policy network until the state value reaches or exceeds the preset threshold. Based on the current momentum coefficient and iteration step size, adversarial examples are generated.

2. The method according to claim 1, characterized in that, The process of converting the image to be attacked into an initial state vector includes: The image is converted into a tensor format that the target model can process, resulting in an image sample tensor. Initialize the current cumulative perturbation tensor to zero; Initialize the normalized iteration step count to zero; Calculate the initial normalized perturbation norm based on the image sample tensor; The image to be attacked is input into the target model, the prediction result is obtained and converted into a one-hot vector, and the image prediction one-hot vector is obtained. The predicted one-hot vector is also used as an adversarial example to predict the one-hot vector; The image tensor, the current cumulative perturbation tensor, the normalized iteration step number, the normalized perturbation norm, the image prediction one-hot vector, and the adversarial example prediction one-hot vector are concatenated to form the initial state vector.

3. The method according to claim 2, characterized in that, The initial state vector is represented as follows: s t =[img t ,pert t ,step t ,norm t ,orig_pred t ,curr_pred t ] Among them, img t For image tensors; pert t This represents the current cumulative perturbation tensor; step t Normalized iteration number; norm t For the normalized perturbation norm; orig_pred t Predict one-hot vectors for an image; curr_pred t To predict one-hot vectors for adversarial examples.

4. The method according to claim 3, characterized in that, Injecting random noise into the current state vector to form a perturbed state vector includes: Random noise with the same dimension as the current accumulated perturbation tensor is sampled from a Gaussian distribution with zero mean and adjustable variance; A random noise matrix is ​​generated based on the sampled random noise and added to the current cumulative perturbation tensor to obtain the cumulative perturbation tensor after injecting noise; The normalized perturbation norm after injecting noise is obtained by recalculating the cumulative perturbation tensor after injecting noise. The cumulative perturbation tensor after injecting noise and the normalized perturbation norm are updated to the current state vector to obtain the perturbed state vector.

5. The method according to claim 1, characterized in that, Iterative training and updating of the parameters of the actor network and the critic network in the policy network includes: The training samples are used as raw adversarial samples and input into the target model to obtain the prediction results of the training samples; An initial state vector is generated based on the training samples and their prediction results, and this initial state vector is used as the current state vector; the following steps are executed iteratively: Step a: Generate action vectors based on the current state vector using the policy network. The action vectors include momentum coefficients and iteration step size. Step b: Execute the MI-FGSM attack update adversarial sample according to the action vector, obtain the next generation state vector according to the updated adversarial sample, and calculate the reward value of this iteration; Repeat step ab using the next-generation state vector as the current state vector until the termination condition is met. Collect the current state vector, reward value, action vector, and next-generation state vector for each iteration as transition data for each iteration; The advantage function estimate is calculated for each iteration based on all transition data, and the parameters of the actor network and the critic network are updated based on the advantage function estimate using a proximal policy optimization algorithm, wherein each iteration includes a preset number of iterations.

6. The method according to claim 5, characterized in that, The estimated value of the advantage function for each iteration, calculated based on the transfer data of each iteration, is shown in the following formula; Where, δ t Let γ be the perturbation in the t-th iteration, λ be the discount factor, λ be the dominance estimation parameter, T be the maximum number of iterations, and δ be the perturbation in the t-th iteration. t+1 For the perturbation in the (t+1)th iteration, δ T-1 The perturbation in the (t-1)th iteration δ t =r t +γV(s t+1 )-V(s t ) Where, r t Let s be the reward value for the t-th iteration. t Let s be the current state vector in the t-th iteration. t+1 Let V(s) be the next state vector for the t-th iteration. t V(s) represents the value estimate of the current state vector of the critic network for the t-th iteration. t+1 ) represents the value estimate of the next state vector of the commentator network for the t-th iteration.

7. The method according to claim 6, characterized in that, Based on the aforementioned advantage function estimate, the parameters of the actor network and the critic network are updated using a proximal policy optimization algorithm, including: Based on the state vectors and action vectors in the collected transition data, calculate the probability ratio of the current policy to the old policy used when collecting the transition data, in generating actions under the same state-action pair. Based on the probability ratio and the advantage function estimate, a pruning objective function is constructed, and the parameters of the actor network are updated by maximizing the pruning objective function; Based on the state vectors and reward values ​​in the collected transition data, a difference objective function is constructed by calculating the difference between the state value output by the critic network and the target value. The parameters of the critic network are updated by minimizing the difference objective function.

8. The method according to claim 7, characterized in that, The reward value is shown in the following formula; Among them, R success R is the reward value for a successful attack. fail The penalty value for a failed attack is I, an indicator function that takes the value 1 if the condition is met, and 0 otherwise; f is the prediction function of the target model, x is the adversarial example, and R is the value of the target model. step Let δ be the iteration efficiency reward coefficient, y be the true label, and δ be the tactical efficiency reward coefficient. t Let ω be the perturbation in the t-th iteration, and let ω be the perturbation penalty coefficient.

9. The method according to claim 8, characterized in that, The termination condition is: The target model predicts an error or the preset maximum number of iterations has been reached.

10. A MI-FGSM adversarial attack system based on reinforcement learning dynamic optimization, characterized in that, include: The training module is used to construct a policy network based on the actor-critic architecture. The parameters of the actor network and the critic network in the policy network are updated through iterative training to obtain the trained policy network. The output module is used to convert the image sample to be attacked into an initial state vector, input the initial state vector into the trained policy network, output the momentum coefficient and iteration step size through the actor network, and evaluate the state value of the current state vector through the critic network. The judgment and generation module is used to generate adversarial samples by executing an attack process based on the current momentum coefficient and iteration step size when the state value reaches or exceeds a preset threshold. If the state value is lower than a preset threshold, random noise is injected into the current state vector to form a perturbed state vector, and the perturbed state vector is re-input into the trained policy network until the state value reaches or exceeds the preset threshold. Based on the current momentum coefficient and iteration step size, the attack process is executed to generate adversarial examples.

Images