Cross-model architecture mobility confrontation sample generation method based on enhanced deformation attack

Abstract

The invention discloses a cross-model architecture transferability confrontation sample generation method based on enhanced deformation attack. The method comprises the following steps: acquiring an input image; the method comprises the following steps: randomly using full grid configuration or central grid configuration for an input image to obtain a plurality of configuration images; for each configuration image, calculating a canvas expansion amount according to a noise scale and generating an expanded canvas image, and obtaining a final deformation image through TPS transformation calculation and cutting operation; randomly selecting an appearance enhancement operation for each final deformed image to generate a final enhanced image; inputting the plurality of enhanced images into a proxy model to obtain a gradient corresponding to a real label so as to calculate an average gradient; based on the average gradient, using a momentum iteration method to update the input image; and repeatedly executing the steps until the maximum number of iterations is reached, and outputting a final adversarial sample. According to the method, through a series of innovative input transformation mechanisms, the limitation of a traditional adversarial sample generation method is broken through, and the generalization ability and attack success rate of adversarial samples are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of adversarial attack technology and relates to a method for generating cross-model architecture transferable adversarial samples based on enhanced deformable attacks. Background Technology

[0002] With the rapid development of deep learning technology, deep neural networks (DNNs) have achieved remarkable results in various fields such as image classification and object detection, and have been widely applied in security-sensitive areas such as autonomous driving, facial recognition, and medical diagnosis. However, the vulnerability of DNNs to adversarial examples has attracted widespread attention. Adversarial examples are generated by adding imperceptible micro-perturbations to normal input data, with the aim of misleading the model to produce incorrect predictions. This vulnerability not only exposes the security risks of the model, but also provides an important direction for studying the robustness and interpretability of the model, making adversarial attack and defense techniques a research hotspot in the field of artificial intelligence security.

[0003] In the field of adversarial attacks, existing methods primarily focus on white-box attack scenarios, where attackers have complete access to the target model's internal structure and parameters. These methods achieve effective attacks on the target model by iteratively updating adversarial perturbations along the gradient direction of the loss function using optimization algorithms. However, in practical applications, attackers often cannot obtain complete information about the target model, making black-box attack scenarios more relevant. In such scenarios, attackers typically generate adversarial examples using a known alternative model and then transfer them to the target model for attack. This attack approach relies on the transferability of the adversarial examples, i.e., their effectiveness across different models.

[0004] Despite existing research efforts aimed at improving the transferability of adversarial examples, current methods still face significant challenges in attack scenarios across different model architectures. For example, the transferability of attacks from convolutional neural networks (CNNs) to visual transformers (ViTs) is poor. This is because CNNs and ViTs have fundamental architectural differences: CNNs capture local texture features through convolutional kernels, while ViTs model global relationships through self-attention mechanisms. This difference leads to the poor performance of adversarial examples generated by existing attack methods when transferring across model architectures, making it difficult to simultaneously influence the decision-making logic of different architectural models.

[0005] Therefore, improving the transferability of adversarial examples across cross-model architectures has become a key research issue. This is not only significant for optimizing adversarial training effects and enhancing the security and generalization capabilities of artificial intelligence models, but also provides a theoretical basis for developing more effective defense strategies.

[0006] Patent "CN121121355A Adversarial Example Generation Method, Apparatus, System, and Storage Medium" enhances the gradient of specific frequency components by designing a frequency domain mask combining low-frequency and mid-frequency masks, and introduces a multi-directional orthogonal direction exploration strategy and an adaptive step size adjustment strategy to generate adversarial examples with high cross-model transferability. Patent "CN120975145A An Adversarial Attack Example Generation Method and Related Equipment" determines the initial value of the adversarial perturbation distribution of the diffusion model and evaluates the perturbation distribution based on a classifier misleading evaluation function to generate adversarial perturbation samples to generate adversarial attack examples. While these patents improve the transferability of adversarial examples by enhancing gradients through frequency domain masks and optimizing the perturbation distribution through diffusion models, there is still room for improvement in their transfer consistency when facing CNN and ViT targets with significant geometric bias differences. Summary of the Invention

[0007] To address the aforementioned technical problems, the present invention aims to provide a cross-model architecture transferable adversarial example generation method based on enhanced morphing attacks, designed to improve the transferability of adversarial examples across different model architectures. This method overcomes the limitations of traditional adversarial example generation methods through a series of innovative input transformation mechanisms, significantly improving the generalization ability and attack success rate of adversarial examples.

[0008] This invention provides a method for generating cross-model architecture transferable adversarial examples based on enhanced morphing attacks, comprising:

[0009] Step 1: Obtain the input image;

[0010] Step 2: Randomly apply either a full-grid configuration or a center-grid configuration to the input image to obtain multiple configuration images;

[0011] Step 3: For each configuration image, calculate the canvas expansion amount based on the noise scale and generate the expanded canvas image. Then, obtain the final deformed image through TPS transformation calculation and cropping operations.

[0012] Step 4: Randomly select an appearance enhancement operation for each final deformed image to generate multiple final enhanced images;

[0013] Step 5: Input multiple enhanced images into the surrogate model to obtain the gradients corresponding to the real labels and then calculate the average gradient;

[0014] Step 6: Update the input image using the momentum iteration method based on the average gradient;

[0015] Step 7: Repeat steps 1-6 until the maximum number of iterations is reached, and output the final adversarial example.

[0016] This invention presents a cross-model architecture transferable adversarial example generation method based on Enhanced Deformation Attack (EDA). Through an innovative input transformation mechanism, it effectively addresses the problem of insufficient transferability due to overfitting in traditional methods during cross-architecture attacks. First, this invention introduces a dual control point layout mechanism, randomly using either a full-mesh configuration or a central mesh configuration, breaking the limitation of fixed edge control points in traditional methods and making the deformation of the entire image region more diverse. Simultaneously, this invention employs an adaptive canvas expansion strategy, calculating the expansion boundary through reflection modes and dynamically adjusting the expansion size based on the maximum potential displacement, thereby creating a semantically consistent buffer at the image edges. This strategy ensures the smoothness and semantic coherence of the deformed image through precise coordinate alignment and Thin Plate Splines (TPS) transformation. Furthermore, this invention combines random appearance enhancement operations, including pixel-level Gaussian noise injection and image-level brightness adjustment, with spatial deformation to increase image diversity and further enrich the perturbation space of adversarial examples.

[0017] Through this series of innovative mechanisms, this invention significantly improves the portability and stability of adversarial examples in cross-model architecture attacks while maintaining computational efficiency, providing important reference and guidance for the research and application of adversarial attack techniques. Attached Figure Description

[0018] Figure 1 This is a flowchart of a cross-model architecture transferability adversarial sample generation method based on enhanced deformation attack according to the present invention;

[0019] Figure 2 This is a schematic diagram of the full grid layout and the central grid layout of the control points;

[0020] Figure 3 It is a comparison display of clean and deformed images. Detailed Implementation

[0021] like Figure 1 As shown, the present invention provides a method for generating cross-model architecture transferable adversarial examples based on enhanced morphing attacks, comprising:

[0022] Step 1: Obtain the input image.

[0023] Step 2: Randomly apply either a full grid configuration or a center grid configuration to the input image to obtain multiple configuration images.

[0024] In image processing and computer vision, Thin Plate Splines (TPS) are a technique for achieving elastic deformation of images. The core idea is to calculate a smooth deformation field by defining a set of control points and their target positions, thereby achieving overall image deformation. The layout and configuration of the control points play a crucial role in TPS deformation. The control points define the local regions and directions of image deformation, and their positions and target offsets determine the overall deformation effect. Specifically, TPS deformation interpolates control points by minimizing bending energy, generating a natural and physically plausible mapping relationship, ensuring that the image retains semantic coherence after deformation. This deformation technique can effectively simulate elastic deformations that may occur in actual image capture, such as bending and stretching, thereby enhancing resistance to transfer through natural deformation.

[0025] Traditional TPS implementations typically fix edge control points to avoid boundary artifacts, but this method limits the degree of freedom of deformation in the central region of the image. This invention introduces a dual control point layout mechanism, which increases the diversity of image deformation by randomly using a full grid (3×3 grid) configuration or a central grid (2×2 grid) configuration.

[0026] The basic representation of a grid is: to create a sparse grid on the input image. Control point grid, forming a control point set ,in Each Representing the normalized space In coordinate.

[0027] In practice, the full-grid configuration uses multiple control points evenly distributed; here we use a 3×3 grid. In the center-grid configuration, the control points are placed at the center of each cell in the full grid, generating a 2×2 center-grid. For example... Figure 2 As shown, in each iteration, a grid configuration is randomly selected, and all control points can move freely within a certain range. This layout strategy, along with random selection to prevent overfitting to a single deformation pattern, effectively improves the generalization ability of adversarial examples.

[0028] Step 3: For each configuration image, adjust according to the noise scale. Calculate the canvas expansion amount and generate the expanded canvas image. The final deformed image is obtained through TPS transformation calculation and cropping operations. This step employs an adaptive canvas expansion strategy, using reflection mode to calculate the expansion boundary. The expansion size is dynamically calculated based on the maximum potential displacement, and artifacts caused by edge control point movement are eliminated by adding semantically consistent buffers at the image edges. The specific operation is as follows:

[0029] Step 3.1: Based on noise scale Calculate the maximum offset Then, based on the configured image size and Calculate the expansion amount of width and height. and :

[0030]

[0031] Step 3.2: Expand the configuration image into a canvas in reflection mode. ,in and This creates a smooth buffer that is sufficient to accommodate the maximum offset of the control point in the next operation.

[0032] Step 3.3: Apply random two-dimensional offset vectors to all control points of the full grid or center grid in the configuration image. Each component of the two-dimensional offset vector is independently derived from... Extracted from the middle, through noise scale To control the deformation intensity, the target deformation point of the control point is... ,in The TPS transformation is performed based on the offset vector to obtain the deformed image.

[0033] Step 3.4: Perform a cropping operation on the deformed image to obtain the final deformed image with the correct size.

[0034] Figure 3 The continuous black borders of the fixed-edge TPS transform (second column) and the effect of the extended method proposed in this invention in eliminating black borders are compared (third column).

[0035] Step 4: For each final deformed image, randomly select an appearance enhancement operation to generate multiple final enhanced images. .

[0036] This invention combines random appearance enhancement with spatial deformation to further enrich the perturbation space for adversarial examples. Specifically, the appearance enhancement operations include pixel-level Gaussian noise injection and image-level brightness adjustment.

[0037] Pixel-level Gaussian noise injection is achieved by injecting a standard deviation of 1 in each RGB channel. Independent Gaussian noise implementation: The default .

[0038] Image-level brightness adjustment via random factor Adjust the overall brightness of the image: In each iteration, one of two transformations is randomly selected and applied to the final deformed image. To obtain an enhanced image :

[0039] .

[0040] These appearance enhancement operations not only increase the diversity of images, complementing structural perturbations, but also maintain our core deformation innovation as the main attack mechanism, thereby improving the process of this invention.

[0041] Step 5: Input multiple enhanced images into the surrogate model to obtain the gradients corresponding to the real labels, and then calculate the average gradient. The surrogate model uses either a CNN model or a Transformer model.

[0042] Step 6: Update the input image based on the average gradient using the momentum iteration method, specifically:

[0043] Step 6.1: Convert the historical momentum vector With the current average gradient The normalized results are summed to obtain the new momentum gradient;

[0044]

[0045] in For momentum gradient, This is the historical momentum vector. The current average gradient, It is the momentum decay factor;

[0046] Step 6.2: Advance the current input image along the direction obtained by the element-wise sign of the momentum gradient by a preset step size to obtain the adjusted image;

[0047]

[0048]

[0049] in, For the input image, To preset the step size, The adjusted image;

[0050] Step 6.3: Crop the pixel values ​​of the adjusted image back to the valid range. This yields the input image for the next step. Budget for maximum disturbance.

[0051] Step 7: Repeat steps 1-6 until the maximum number of iterations is reached, and output the final adversarial example.

[0052] Through this series of innovative input transformation mechanisms, the method of this invention significantly improves the transferability and stability of adversarial examples in cross-model architecture attacks while maintaining computational efficiency, providing new ideas and methods for the development of adversarial attack technology.

[0053] Example

[0054] Experiments were conducted on an ImageNet-compatible dataset, which has been widely used in previous research. This dataset contains 1000 images of size 299×299×3, covering 1000 categories, with each image labeled with both ground truth and target attack labels. Experiments involved various source and target models, including CNN architectures such as ResNet-18, Inception-v3, Inception-v4, and Inception-ResNet-v2, as well as Transformer architectures such as ViT-Base, DeiT-Base, and LeViT. Furthermore, various adversarial training-enhanced defense models were tested to evaluate the effectiveness of EDA against complex defense mechanisms. Five popular input transformation-based iterative adversarial attack methods were selected as baselines, including L2T, BSR, DeCoWA, OPS, and SID.

[0055] To ensure the comparability and reliability of the experimental results, all attack methods followed a unified and general setting. Specifically, the generation of adversarial examples was based on the momentum iterative method, with a maximum perturbation budget. Set as Number of iterations For 10 repetitions, step size The momentum decay factor is 1.6. The parameter value is 1. These parameters were chosen to simulate common constraints in real-world attack scenarios while ensuring a fair comparison between attack methods. Attack success rate (ASR) was used to quantify attack performance. Experiments were conducted on a single NVIDIA RTX 3090 GPU.

[0056] In the specific attack method design, the EDA method of this invention introduces a unique input transformation mechanism to enhance the transferability of adversarial examples. EDA noise scale. Setting it to 0.55 controls the intensity of elastic deformation, ensuring that semantic consistency of the image is maintained while increasing deformation diversity. The L2T setting remains the same as the original setting. According to the OPS notation, OPS... This indicates that each iteration is performed. Each iteration uses... Operator samples and A total of [number] perturbation samples were processed. Subgradient calculation. To match the sampling complexity of OPS(10, 5, 5), for The gradients of BSR, DeCoWA, SID, and the proposed EDA method are calculated for the transformed image, while keeping the original settings of all other hyperparameters unchanged to ensure the fairness of the experimental results.

[0057] As shown in Table 1, adversarial transferability in CNN→CNN co-source scenarios (using CNN proxy models to attack various CNN target models) was evaluated, and the EDA method proposed in this invention demonstrated excellent performance. When using Inc-v3 as the source model, EDA achieved the best attack success rate on all black-box models compared to other methods. This consistently superior performance trend was reflected in all source models, including ResNet-18, Inception-v4, and Inception-ResNet-v2, where EDA achieved the highest average attack success rate. The results confirm that even in co-source attack scenarios, EDA can generate highly transferable samples.

[0058] Table 1 shows the success rate (%) of attacks against ten CNN models on the ImageNet-compatible dataset. In the table, * indicates a white-box model, and no * indicates a black-box model.

[0059]

[0060] In the more challenging CNN→ViT cross-model family scenario, the advantages of EDA are further amplified. Our method consistently outperforms all baseline methods across all source model configurations and ten target ViT architectures. With ResNet-18 as the source model, the attack success rate reaches 71.7% against ViT-Base, 85.3% against DeiT-Base, 88.5% against Swin-T, and 92.8% against CoaT-T, comprehensively surpassing the comparison methods. With Inception-v3 as the source model, EDA achieves a success rate of 58.0% against ViT-Base, 73.9% against DeiT-Base, 74.3% against Swin-T, and 84.0% against CoaT-T. Even with significant differences in the receptive field and feature distribution of the source models, it maintains consistent suppression of the ViT family, demonstrating that its variant strategy has universal destructive power against various self-attention mechanisms. Similar trends were replicated in experiments using Inception-v4 and Inception-ResNet-v2 as sources, demonstrating stable cross-architecture generalization capabilities.

[0061] Table 2 shows the attack success rate (%) for ten ViT models on the ImageNet compatible dataset. * indicates a white-box model, and no * indicates a black-box model.

[0062]

[0063] We evaluated EDA using ViT-B and LeViT as source models to test its generalization ability with ViT as the surrogate model. To verify EDA's real-world transferability across architectures, the experiments were conducted sequentially along two paths: "ViT→ViT" and "ViT→CNN". First, using ViT-B and LeViT as source models, we launched black-box attacks directly on the 10 ViT target models in Table 1 without any retraining or parameter fine-tuning. As shown in Table 3, EDA achieved an average attack success rate of 93.5% (ViT-B source) and 96.7% (LeViT source) on the ViT→ViT path, with a difference of no more than 0.3 percentage points from the best SID method, indicating that our method maintains high transferability across similar global attention architectures. Subsequently, maintaining identical hyperparameters, we applied adversarial examples generated from the same batch of ViT source models to the 10 CNN target models in Table 2. EDA achieved average success rates of 87.7% and 96.7% on the ViT→CNN link, outperforming the second-place SID by up to 0.3 percentage points and matching the best baseline at its lowest. These results confirm that EDA remains competitive even when using ViT-based alternative models.

[0064] Table 3 shows the average attack success rate (%) for ten CNN models and ten ViT models on the ImageNet compatible dataset.

[0065]

[0066] The experiments progressed sequentially, without additional tuning of the target model or introduction of learnable parameters, yet maintained top- or near-top-tier transfer performance on both heterogeneous paths simultaneously. This objectively demonstrates that EDA can achieve consistent attacks across model architectures simply through a unified geometry-appearance composite perturbation. In summary, the method of this invention significantly improves the transfer attack capability of adversarial examples under cross-model mechanisms, verifying the effectiveness and practical value of this method in the field of adversarial transferability enhancement.

[0067] The above description is only a preferred embodiment of the present invention and is not intended to limit the ideas of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for generating cross-model architecture transferable adversarial examples based on enhanced deformation attacks, characterized in that, include: Step 1: Obtain the input image; Step 2: Randomly apply either a full-grid configuration or a center-grid configuration to the input image to obtain multiple configuration images; Step 3: For each configuration image, adjust according to the noise scale. Calculate the canvas expansion amount and generate the expanded canvas image. The final deformed image is obtained through TPS transformation calculation and cropping operations. ; Step 4: For each final deformed image, randomly select an appearance enhancement operation to generate multiple final enhanced images. ; Step 5: Input multiple enhanced images into the surrogate model to obtain the gradients corresponding to the real labels and then calculate the average gradient; Step 6: Update the input image using the momentum iteration method based on the average gradient; Step 7: Repeat steps 1-6 until the maximum number of iterations is reached, and output the final adversarial example.

2. The method for generating cross-model architecture transferable adversarial examples based on enhanced morphing attacks according to claim 1, characterized in that, In step 2, the full-mesh configuration uses multiple control points evenly distributed, while the central-mesh configuration places the control points at the center of each cell in the full mesh. The basic representation of the mesh is: Create a sparse array on the input image. Control point grid, forming a control point set ,in Each Representing the normalized space In coordinate.

3. The method for generating cross-model architecture transferable adversarial examples based on enhanced morphing attacks according to claim 1, characterized in that, Step 3 specifically involves: Step 3.1: Based on noise scale Calculate the maximum offset Then, based on the configured image size and Calculate the expansion amount of width and height. and : Step 3.2: Expand the configuration image into a canvas in reflection mode. ,in and This creates a smooth buffer that is sufficient to accommodate the maximum offset of the control point in the next operation; Step 3.3: Apply random two-dimensional offset vectors to all control points of the full grid or center grid in the configuration image. Each component of the two-dimensional offset vector is independently derived from... Extracted from the middle, through noise scale To control the deformation intensity, the target deformation point of the control point is... ,in ; The TPS transformation is performed based on the offset vector to obtain the deformed image; Step 3.4: Perform a cropping operation on the deformed image to obtain the final deformed image with the correct size.

4. The method for generating cross-model architecture transferable adversarial examples based on enhanced morphing attacks according to claim 1, characterized in that, The appearance enhancement operations in step 4 include pixel-level Gaussian noise injection and image-level brightness adjustment.

5. The method for generating cross-model architecture transferable adversarial examples based on enhanced morphing attacks according to claim 4, characterized in that, Pixel-level Gaussian noise injection is achieved by injecting a standard deviation of 1 in each RGB channel. Independent Gaussian noise implementation: The default .

6. The method for generating cross-model architecture transferable adversarial examples based on enhanced morphing attacks according to claim 5, characterized in that, Image-level brightness adjustment via random factor To adjust the overall brightness of the image: In each iteration, one of two transformations is randomly selected and applied to the final deformed image. To obtain an enhanced image : 。 7. The method for generating cross-model architecture transferable adversarial examples based on enhanced morphing attacks according to claim 1, characterized in that, The proxy model uses either a CNN model or a Transformer model.

8. The method for generating cross-model architecture transferable adversarial examples based on enhanced morphing attacks according to claim 1, characterized in that, Step 6 specifically involves: Step 6.1: Convert the historical momentum vector With the current average gradient The normalized results are summed to obtain the new momentum gradient; in For momentum gradient, This is the historical momentum vector. The current average gradient, It is the momentum decay factor; Step 6.2: Advance the current input image along the direction obtained by the element-wise sign of the momentum gradient by a preset step size to obtain the adjusted image; in, For the input image, To preset the step size, The adjusted image; Step 6.3: Adjust the image Cropping pixel values ​​back to the valid range This yields the input image for the next step. Budget for maximum disturbance.

Images