Vision-based model oriented domain adaptation adversarial attack method and system

Abstract

The present application belongs to the technical field of computer vision and attack and defense resistance, and provides a domain adaptation attack method and system for a visual basic model, and proposes an attack sample generation method capable of migrating attacks on unknown fine-tuning models by using an open-source basic model under a black-box condition. The attack process consists of two stages of offline domain alignment initialization and online instance-level adaptation. In the offline stage, the basic model is used to mine general vulnerabilities on a proxy data set to generate domain-aligned general perturbations; in the online stage, the general perturbations are introduced into the optimization process through momentum hot start, the structural gradient alignment module is used to guide the attack to focus on significant boundaries, and the amplitude deviation simulation module is combined to simulate the feature drift caused by fine-tuning in the frequency domain. The problems of domain offset and parameter difference between the open-source basic model and the specific domain model after fine-tuning are solved.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision and attack and defense technology, and in particular relates to a domain-adaptive adversarial attack method and system for vision basic models. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] In recent years, large-scale vision foundational models, represented by SAM (Signal-Based Amplifier), have revolutionized the field of image segmentation. To apply SAM's powerful generalization capabilities to specific vertical domains, researchers have proposed various efficient parameter fine-tuning schemes. For example, Wu et al.'s Medical SAM Adapter, by inserting an adapter layer into the encoder, enables SAM to adapt to specific domain segmentation tasks such as medical imaging; Zhang et al.'s SAMed utilizes LoRA technology for low-rank updates of SAM's attention mechanism. While this paradigm significantly improves model performance on specific tasks, the public nature of its backbone network parameters provides black-box attackers with a natural "proxy model." Adversarial attacks aim to exploit the vulnerabilities of deep learning models; in security-critical scenarios, malicious adversarial examples can lead to serious consequences. Therefore, evaluating the robustness of the fine-tuned model is a necessary step before model deployment.

[0004] Existing black-box migration attack techniques are mainly based on gradient optimization and feature perturbation, and can be roughly divided into the following categories: (1) Gradient-based momentum optimization methods: represented by MI-FGSM proposed by Dong et al. This method introduces a momentum term during the iterative generation of adversarial examples, stabilizing the update process by accumulating previous gradient directions, avoiding attacks from getting trapped in local maxima, and thus improving the transferability of adversarial examples. The subsequent Nesterov Accelerated Gradient (NI-FGSM) further optimized this process. These methods are currently the most common baseline attack techniques.

[0005] (2) Input transformation enhancement methods: represented by DMI proposed by Xie et al. and ANDA proposed by Fang et al. These methods simulate inputs of different scales and perspectives by randomly scaling, padding, and blending clean input images, aiming to generate robust perturbations that are insensitive to model transformations.

[0006] Building upon these general adversarial attack methods, the recent popularity of SAM (Sensitive Image Classification) has led to various attacks targeting SAM. For example, Zhang et al. proposed Attack-SAM, which attempts to attack the mask decoder of SAM; DarkSAM achieves its attack by combining spatial and frequency domain information to interfere with the intermediate layer features of the image encoder. Furthermore, Xia et al. attempted to utilize a meta-learning framework to improve the transferability of attacks against downstream SAM tasks. While these methods perform reasonably well on natural image classification tasks or the original SAM model, their effectiveness is not significant when attacking "fine-tuned domain-specific SAM models."

[0007] The inventors discovered that the current method still exists: (1) Ignoring the “spectral drift” caused by fine-tuning: The fine-tuning module is essentially a reshaping of the model’s feature space. Existing methods assume that the feature responses of the source model and the target model are consistent, completely ignoring the physical law that fine-tuning mainly changes the amplitude spectrum of the features while relatively preserving the phase spectrum. This leads to the generated attack being prone to overfitting the texture features of the original SAM. Once the texture features are changed by fine-tuning, the attack fails. (2) Lack of “structure awareness” in a specific domain: Existing attacks usually distribute the perturbation energy evenly or concentrate it in texture-rich regions. However, in medical image segmentation or camouflage targets, the targets have extremely strong structural consistency and boundary sensitivity. Existing methods fail to utilize this structural prior of a specific domain, resulting in the perturbation failing to concentrate on attacking the model’s most vulnerable boundary discrimination ability. (3) Blindness of initialization: Existing methods usually start optimization from zero (zero vector) or random noise. Since the data manifold distance between the source domain and the target domain is extremely large, blind initialization makes it difficult for the optimization process to cross the domain gap and converge to an adversarial subspace that is general to the target domain. Summary of the Invention

[0008] To address at least one of the technical problems mentioned above, this invention provides a domain-adaptive adversarial attack method and system for visual basic models, proposing a black-box attack scheme that can overcome the aforementioned dual differences and achieve efficient attack migration without obtaining the target model parameters.

[0009] To achieve the above objectives, the present invention adopts the following technical solution: The first aspect of the present invention provides a domain adaptation adversarial attack method for visual fundamental models, comprising the following steps: Offline domain alignment initialization phase: Construct a proxy dataset with a distribution similar to the target domain, and optimize the generation of general adversarial perturbations on the proxy dataset; Online instance-level adaptive phase: For a single clean input image, the general adversarial perturbation is transformed into initial momentum for online iterative optimization, and the following iterative optimization process is performed: Extract the structural prior map from the clean input image, and use the structural prior map as a guiding mask to modulate the attack gradient to obtain the corrected attack gradient; In the frequency domain, simulated fine-tuning of the original features of the visual basic model causes feature drift, generating simulated bias features; A composite loss function is constructed based on the features of adversarial examples, the original features of the input clean image, and the simulated bias features. The attack gradient flow direction and the initial state are deeply reconstructed by the momentum iteration algorithm in combination with the composite loss function until the final adversarial example is generated.

[0010] Furthermore, the optimization objective of the general perturbation is to maximize the expected feature distortion of the base model encoder on the proxy dataset, which is mathematically expressed as: ， in, This represents the optimized general adversarial perturbation. Indicates in the proxy dataset Above is a clean input image Take the math period, An image encoder representing the fundamental model of vision. express Norm distance measures the degree of distortion in the feature space. This represents a candidate adversarial perturbation in the optimization process, and its value range is subject to a predefined perturbation bound. Constraints.

[0011] Furthermore, the calculation process of using the structural prior graph as a guiding mask to modulate the attack gradient is as follows: , in, No. t The attack gradient after the next iteration. It is the first t The original loss gradient calculated in the next iteration It is a scaling factor that controls the structural reinforcement strength. This indicates element-wise multiplication. Represents the structural prior diagram.

[0012] Furthermore, the simulated feature drift caused by fine-tuning the original features of the visual base model in the frequency domain, generating simulated bias features, includes: The original features of the base model encoder are decomposed in the frequency domain using fast Fourier transform to obtain the phase spectrum and amplitude spectrum of the input clean image. Construct a spectral uncertainty profile that is proportional to the amplitude spectrum to capture potential feature drift; The random noise matrix following a standard Gaussian distribution is modulated using spectral uncertainty profiles, and the simulated bias characteristics are obtained through inverse fast Fourier transform.

[0013] Furthermore, in the parameter update during the offline domain alignment initialization phase, a sign-based momentum gradient descent method is adopted, in the... In the offline iteration, the mathematical update formula for the general adversarial perturbation is: , in, Indicates the first The general adversarial perturbation updated offline. Indicates the first General adversarial perturbation for offline iteration. The operator strictly constrains the obtained perturbation value to a pre-defined infinite norm boundary. Within the range, The set offline optimization step size, The function extracts the sign and direction of the gradient to ensure the stability of the update. For offline loss function, This represents the gradient operator of the offline loss function with respect to the perturbation variable.

[0014] Furthermore, the composite loss function is: , in, Indicates the current number Features extracted from the visual base model based on the adversarial examples generated in each iteration. To input a clean image The original characteristics, In the first The feature drift is dynamically generated step-by-step to simulate the effect of unknown fine-tuning parameters. Indicates the first The instance-level adversarial perturbation is generated through step-by-step online iteration.

[0015] Furthermore, the combined composite loss function performs a deep reconstruction of the attack gradient flow direction and initial state using a momentum iteration algorithm, including: Using the domain-wide perturbation extracted in the offline phase, a hot-start initialization of the online momentum buffer is performed. In each iteration, the original gradient with respect to the image is calculated for the composite loss function; The original gradient is spatially modulated using the structural prior graph to strengthen the edge attack weights, and the accumulated momentum is updated using the modulated gradient; the actual perturbation increment is calculated based on the updated accumulated momentum.

[0016] A second aspect of the present invention provides a domain-adaptive adversarial attack system for visual fundamental models, comprising: The offline domain alignment initialization module is used to construct a proxy dataset with a distribution similar to the target domain, and to optimize the generation of general adversarial perturbations on the proxy dataset. The online instance-level adaptive phase module is used to transform general adversarial perturbations into initial momentum for online iterative optimization for a single clean input image, and performs the following iterative optimization process: Extract the structural prior map from the clean input image, and use the structural prior map as a guiding mask to modulate the attack gradient to obtain the corrected attack gradient; In the frequency domain, simulated fine-tuning of the original features of the visual basic model causes feature drift, generating simulated bias features; A composite loss function is constructed based on the features of adversarial examples, the original features of the input clean image, and the simulated bias features. The attack gradient flow direction and the initial state are deeply reconstructed by the momentum iteration algorithm in combination with the composite loss function until the final adversarial example is generated.

[0017] A third aspect of the present invention provides a computer-readable storage medium.

[0018] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the domain adaptation adversarial attack method for a vision-based model as described above.

[0019] A fourth aspect of the present invention provides a computer device.

[0020] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the domain adaptation adversarial attack method for vision-based models described above.

[0021] Compared with the prior art, the beneficial effects of the present invention are: This invention overcomes the "feature space parameter drift" barrier introduced by model fine-tuning. Most existing state-of-the-art black-box feature layer attack methods assume that the source and target models have highly consistent feature activation spaces, generating adversarial examples by blindly perturbing the static features of the source model. However, fine-tuning techniques such as LoRA and Adapter fundamentally reshape the feature space. This invention, through an innovative "amplitude deviation simulation" module, proactively simulates this parameter drift in the frequency domain under black-box conditions without obtaining any target model weights. This allows the attack algorithm to generalize to unknown fine-tuned models, achieving a cross-model transfer attack success rate far exceeding existing technologies.

[0022] This invention eliminates the attack blind spot caused by domain offset, achieving precise destruction of structural boundaries. Existing methods typically rely on prior knowledge of natural images for blind searching or uniformly distributing perturbation energy, resulting in inefficiency when dealing with medical or industrial images with strong structural consistency. This invention not only provides accurate domain knowledge hot-start through optimization of proxy datasets in the "offline stage," but also precisely focuses limited perturbation energy onto the semantic boundaries of the target through a "structural gradient alignment" mechanism. This allows the generated adversarial examples to induce severe structural segmentation collapse in the target segmentation model with smaller perturbation amplitude and more visually concealed characteristics.

[0023] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0024] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0025] Figure 1 This is a flowchart of the domain adaptation adversarial attack method for visual basic models provided in this embodiment of the invention; Figure 2 These are the SAM segmentation visualization examples on the BTCV dataset provided in this embodiment of the invention; where (a) is the clean input image, (b) is the adversarial sample obtained in this experiment, (c) is the ground truth result, and (d) is the adversarial sample segmentation result; Figure 3 These are the SAM segmentation visualization examples on the ISIC dataset provided in this embodiment of the invention; where (a) is the clean input image, (b) is the adversarial sample obtained in this experiment, (c) is the ground truth result, and (d) is the adversarial sample segmentation result; Figure 4 These are the SAM segmentation visualization examples on the REFUGE dataset provided in this embodiment of the invention; where (a) is the clean input image, (b) is the adversarial sample obtained in this experiment, (c) is the ground truth result, and (d) is the adversarial sample segmentation result. Detailed Implementation

[0026] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0027] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0028] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0029] The Segment Anything Model (SAM) is a visual foundational model proposed by Meta AI and pre-trained on a dataset of billions of masks.

[0030] Fine-tuning refers to the process of adapting a model to a specific downstream task by inserting parameter fine-tuning modules while keeping most of the parameters of the base model frozen.

[0031] Adversarial attacks are techniques that introduce minute perturbations into the input data that are imperceptible to the human eye, inducing deep neural networks to produce erroneous outputs.

[0032] Domain-adaptive adversarial attacks refer to the process of transferring attack patterns learned from the natural image domain to specific data domains such as medicine and industry, or to downstream tasks.

[0033] In the process of realizing this invention, the inventors discovered that the prior art has at least the following drawbacks and deficiencies: (1) Low attack mobility: Under the dual challenges of cross-model and cross-data domain, the adversarial examples generated by existing methods are difficult to cause substantial segmentation errors in the target model, and the Dice coefficient does not decrease significantly. (2) Redundancy of adversarial noise: Existing methods often need to add a large amount of noise across the entire image to be effective, which not only reduces the stealth but also fails to accurately target key features. (3) Lack of modeling for fine-tuning mechanisms: The failure to conduct targeted simulations of parameter changes introduced by fine-tuning structures such as LoRA or Adapter leads to a disconnect between the theoretical level of the attack algorithm and the actual fine-tuning scenario.

[0034] This invention designs a two-stage adversarial attack architecture that effectively overcomes domain shift and parameter discrepancies, fully combining the advantages of offline domain prior mining and online instance-specific adaptation. Based on the traditional transfer attack framework, an improved method called "Domain Aligned Initialization and Instance-specific Adaptation (DIIA)" is proposed. First, an offline domain alignment initialization mechanism is constructed. By mining general adversarial perturbations on proxy datasets and transforming them into momentum-based hot-start, the problem of inconsistent distribution between the source domain natural image and the target domain specific image is solved, ensuring the attack starts in the correct direction. Second, frequency domain analysis and gradient optimization are combined to design an Amplitude Deviation Simulation (ADS) module. This module, without obtaining fine-tuning parameters, simulates the feature space drift caused by fine-tuning by locking the phase spectrum and perturbing the amplitude spectrum, forcing the generated adversarial examples to destroy robust structural features that remain unchanged before and after fine-tuning. In addition, a structural gradient alignment (SGA) strategy was designed to locate salient edges in the image using gradient information and concentrate the attack firepower on these key contours to maximize the destruction of the boundary prediction ability of the segmentation model, ultimately achieving a highly destructive black-box attack with strong generalization ability.

[0035] Example 1 like Figure 1 As shown, this invention proposes a method for generating adversarial examples under black-box conditions using an open-source base model, capable of transferring attacks to unknown fine-tuned models. The attack process consists of two stages: offline domain alignment initialization and online instance-level adaptation. In the offline stage, the base model is used to mine general vulnerabilities on a proxy dataset to generate general perturbations for domain alignment. In the online stage, the general perturbations are introduced into the optimization process through momentum hot-start, the Structural Gradient Alignment (SGA) module is used to guide the attack to focus on salient boundaries, and the Amplitude Deviation Simulation (ADS) module is combined to simulate feature drift caused by fine-tuning in the frequency domain.

[0036] This embodiment provides a domain adaptation adversarial attack method for visual basic models, including the following steps: Step 1: Offline Domain Alignment Initialization Phase: Construct a proxy dataset with a distribution similar to the target domain, and optimize the generation of general adversarial perturbations on the proxy dataset; When attacking fine-tuned domain-specific models, directly using adversarial examples generated from natural images often proves ineffective because domain-specific models have unique textures and highly consistent structural patterns.

[0037] To overcome this domain bias, this embodiment extracts common knowledge of a specific domain through offline optimization. First, a proxy dataset with a distribution similar to the target domain is constructed. This study aims to uncover the general vulnerability of image sharing in this domain. It assumes that although images within a specific domain differ in content, they share underlying structural commonalities, leading to shared "blind spots" in the image encoder of the base model when processing these images. Therefore, a general adversarial perturbation is optimized on a proxy dataset. The optimization objective of this general perturbation is to maximize the expected feature distortion of the base model encoder on the proxy dataset, which is mathematically expressed as: , in, This represents the optimized general adversarial perturbation. Indicates in the proxy dataset Above is a clean input image Take the math period, An image encoder representing the fundamental model of vision. express Norm distance measures the degree of distortion in the feature space. This represents a candidate adversarial perturbation in the optimization process, and its value range is subject to a predefined perturbation bound. Constraints (i.e.) ).

[0038] Through this process It converges to a robust direction, capturing the global gradient prior of the domain, and provides a domain-aligned initialization basis for subsequent online attacks.

[0039] Step 2: Online instance-level adaptive phase: For a single clean input image, a general perturbation is introduced into the optimization process through momentum hot start, the Structural Gradient Alignment (SGA) module is used to guide the attack to focus on significant boundaries, and the Amplitude Deviation Simulation (ADS) module is combined to simulate the feature drift caused by fine-tuning in the frequency domain. For a specific single clean input image When launching an attack, simply superimposing a general perturbation may cause the attack to fail due to spatial mismatch. Furthermore, parameter changes introduced during fine-tuning can lead to drift in the feature space. Therefore, this invention designs an instance-level adaptive process in the online phase, specifically including the following steps: Step 201: Transform the general adversarial perturbation into an initial momentum for online iterative optimization; In order to effectively transfer domain-specific knowledge learned offline to online attacks, this invention does not... Instead of using it directly as additive noise, it is transformed into the initial momentum direction for online iterative optimization. Specifically, it will... Perform normalization and initialize the momentum buffer. : , in, This represents the L1 norm. This transformation mechanism allows domain knowledge learned offline to serve as "hot-start" momentum, guiding the optimization trajectory of the online attack. This means that in the first iteration, the attack algorithm's search direction is already in a region that poses a high threat to the specific domain encoder, thus avoiding getting trapped in local optima for natural images.

[0040] Step 202: Extract the structure prior map of the input clean image, and use the structure prior map as a guiding mask to modulate the attack gradient to obtain the corrected attack gradient; A significant difference between domain-specific images and natural images is that their key semantic information is highly dependent on the structural boundaries of the target object. To disrupt the model's ability to perceive these key structures, this invention proposes a structural gradient alignment module. First, the structural prior of the image needs to be extracted. This is achieved by calculating the input clean image... Pixel-level gradient magnitudes are used to construct a structural prior map. : , in, and These represent the spatial gradients of the input clean image in the horizontal and vertical directions, respectively. This represents a normalization operation, mapping values ​​to the [0,1] interval to ensure that the mask weights are within a reasonable range. The prominent boundaries in the image are highlighted, as these areas are the most sensitive parts for the segmentation model.

[0041] Next, during the iterative update process of online attacks, utilizing... Used as a bootstrap mask to modulate the attack gradient. Modified attack gradient. The calculation process is as follows: , in, No. t The attack gradient after the next iteration. It is the first t The original loss gradient calculated in the next iteration It is a scaling factor that controls the structural reinforcement strength. This indicates element-wise multiplication.

[0042] Step 203: Simulate the feature drift caused by fine-tuning the sharp features of the visual basic model in the frequency domain to generate simulated deviation features; This is the core innovative module of the present invention that solves the parameter differences caused by fine-tuning. Fine-tuning technology changes the behavior of the model by inserting a small number of parameters. Because attackers are in a black-box environment, they cannot obtain the fine-tuning parameter increments of the target model, and directly attacking the source model often results in failure to transfer the model.

[0043] This invention is based on a spectral parametric decomposition assumption: the parameter increments introduced by fine-tuning do not randomly alter features, but rather possess spectral bias characteristics. Specifically, the image topology corresponding to the phase spectrum is a task-consistent domain-invariant feature, while the texture and style responses corresponding to the amplitude spectrum are domain-specific and are the main components altered by fine-tuning. Therefore, fine-tuning primarily causes a drift in the feature response across the amplitude spectrum. To simulate this unknown feature drift on the source model, this embodiment incorporates an enhancement operation in the frequency domain.

[0044] Specifically, the steps include the following: Step 2031: Extract clear features from the base model encoder Frequency domain decomposition is performed using Fast Fourier Transform (FFT) to obtain the phase and amplitude features of the clean input image: , , in, Frequency domain decomposition results representing clear features, The amplitude spectrum represents the distribution information of feature activation intensity and texture. This represents the phase spectrum, which contains information about the spatial structure's location.

[0045] Step 2032: Construct the amplitude uncertainty bias; By constructing a spectral uncertainty profile proportional to the amplitude spectrum Used to capture potential feature drift; In this embodiment, the magnitude of the drift is considered to be related to the intensity of the original feature activation, expressed as: , in, It is a very small constant used to ensure numerical stability; It is an adaptive scaling factor that dynamically adjusts the perturbation intensity according to the current attack progress. Its calculation is based on the distance between the current adversarial feature and the original feature. , in, This represents the characteristics of the adversarial examples generated in the previous iteration. This dynamic adjustment mechanism ensures that the simulation bias gradually becomes more refined as the attack progresses.

[0046] Step 2033: Inject the amplitude uncertainty bias and reconstruct the features to obtain the simulated bias features; With the original phase spectrum locked, a random noise matrix following a standard Gaussian distribution is sampled. The spectral uncertainty profile was modulated, and the simulated bias characteristics were obtained through inverse fast Fourier transform (IFFT). : , in, It is the imaginary unit. This represents an efficient mathematical approximation of the drift of unknown fine-tuning parameters of the target model in a black-box state.

[0047] Step 3: Construct the loss function and optimization strategy to optimize the offline and online stages to obtain the final adversarial optimization sample; This invention does not "train" any neural network weights in the online phase, but instead performs "joint iterative optimization" on adversarial perturbations of a single clean input image. To achieve cross-domain and cross-model attack transfer, the optimization strategy of this invention is strictly divided into two progressively related phases: offline and online.

[0048] The only optimization variable in the offline phase is the general adversarial perturbation. Its core objective is to maximize the proxy dataset. In the basic model image encoder In Normative characteristic distortion.

[0049] In practical engineering optimization, small batches (denoted as ) are used. Data sampling is used to approximate the expectation, and an offline loss function is employed. The precise form of is defined as: , In offline parameter updates, a sign-based momentum gradient descent method is used, at the... In the offline iteration, the mathematical update formula for the perturbation is: , in, Indicates the first The general adversarial perturbation updated offline. Indicates the first General adversarial perturbation for offline iteration. This represents the gradient operator of the offline loss function with respect to the perturbation variable. The set offline optimization step size; The function extracts the sign and direction of the gradient to ensure the stability of the update; The operator strictly constrains the obtained perturbation value to a pre-defined infinite norm boundary. Within the range. Passing through. After sufficient iterations, the final output is This serves as the input parameter for the online phase.

[0050] The optimization target in the online phase is a specific single clean input image. Instance-level adversarial perturbation This phase will be conducted offline prior. Frequency domain deviation characteristics generated by the ADS module and the structural prior mask generated by the SGA module Integrate and perform joint computing.

[0051] Traditional feature-layer adversarial attacks typically aim to maximize the Euclidean or cosine distance between the adversarial sample features and the original features of the source model. However, this strategy is prone to overfitting the static feature space of the source model when facing fine-tuned models. This invention proposes a dynamic composite adversarial objective, whose core loss function is... Defined as: , in, Indicates the current number Features extracted from the visual base model from the adversarial examples generated in each iteration; Indicates the first Step-by-step online iterative accumulation of instance-level adversarial perturbations To input a clean image The original characteristics; For the ADS module in the The feature drift is dynamically generated step-by-step and used to simulate the amount of feature drift caused by unknown fine-tuning parameters. The physical meaning of this loss function is that the attack algorithm no longer blindly deviates from the original features, but is forced to maximize the deviation from the "composite features that cover the fine-tuning simulation bias". This allows the generated adversarial perturbations to effectively destroy those robust structural features that remain consistent across domains before and after fine-tuning, thereby generating a very strong generalization attack capability against unknown fine-tuning target models.

[0052] In a single iteration, this invention incorporates the framework of the Momentum Iteration Algorithm (MI-FGSM), but deeply reconstructs the gradient flow direction and the initial state of the online-optimized momentum buffer of this conventional algorithm. First, it utilizes the domain-wide perturbation extracted in the offline phase. Perform a warm-start initialization of the online momentum buffer. In the... In the iteration, the system first targets the composite loss function. Calculate the original gradient with respect to the input clean image. : , Subsequently, the original gradient is spatially modulated using the structural prior mask map generated by the SGA module. After strengthening the edge attack weights, the modulated gradient is then used. Update cumulative momentum To stabilize the optimized trajectory and avoid local maxima: , in, It is a hyperparameter representing the momentum decay factor.

[0053] Finally, the actual disturbance increment is calculated based on the updated momentum, and the amplitude is truncated: , in, This refers to the single-step attack step length. The function extracts the sign and direction of momentum; The operator ensures that the accumulated perturbation is strictly limited to a preset concealment limit. Inside. After After several iterations, the final output is the adversary. .

[0054] The above method overcomes the "feature space parameter drift" barrier introduced by model fine-tuning. Most existing state-of-the-art black-box feature layer attack methods assume that the source and target models have highly consistent feature activation spaces, generating adversarial examples by blindly perturbing the static features of the source model. However, fine-tuning techniques such as LoRA and Adapter fundamentally reshape the feature space. This invention, through an innovative "amplitude deviation simulation" module, proactively simulates this parameter drift in the frequency domain under black-box conditions without obtaining any target model weights. This allows the attack algorithm to generalize to unknown fine-tuned models, achieving a cross-model transfer attack success rate far exceeding existing technologies.

[0055] The method of this invention eliminates the attack blind zone caused by domain offset, achieving precise destruction of structural boundaries. Existing methods typically rely on prior knowledge of natural images for blind searching or uniformly distributing perturbation energy, resulting in inefficiency when dealing with medical or industrial images with strong structural consistency. This invention not only provides accurate domain knowledge hot-start through optimization of proxy datasets in the "offline stage," but also precisely focuses limited perturbation energy onto the semantic boundaries of the target through a "structural gradient alignment" mechanism. This allows the generated adversarial examples to induce severe structural segmentation collapse in the target segmentation model with smaller perturbation amplitude and more visually concealed characteristics.

[0056] Using a medical dataset as an example for verification, the method of this invention can be extended to other specific domains. Extensive experiments have shown that, for example... Figure 2 The image shows the visualization results of SAM segmentation on the BTCV dataset; where (a) is the clean input image, (b) is the adversarial example obtained in this experiment, (c) is the ground truth result, and (d) is the adversarial example segmentation result; as shown... Figure 3 The image shows the visualization results of SAM segmentation on the ISIC dataset; where (a) is the clean input image, (b) is the adversarial example obtained in this experiment, (c) is the ground truth result, and (d) is the adversarial example segmentation result; Figure 4 Visualization results of SAM segmentation on the REFUGE dataset; where (a) is the clean input image, (b) is the adversarial example obtained in this experiment, (c) is the ground truth result, and (d) is the adversarial example segmentation result; it shows that the DIIA method of this invention significantly outperforms state-of-the-art transfer attack methods in both qualitative and quantitative metrics. From a qualitative visualization perspective, the DIIA method of this invention generates results that maintain extremely high visual concealment while causing the model output to produce completely collapsed masks with large areas of blank space or severe distortion, demonstrating the superior ability of this invention to disrupt the fine-tuning model in real-world scenarios.

[0057] Example 2 This embodiment provides a domain-adaptive adversarial attack system for visual basic models, including: The offline domain alignment initialization module is used to construct a proxy dataset with a distribution similar to the target domain, and to optimize the generation of general adversarial perturbations on the proxy dataset. The online instance-level adaptive phase module is used to transform general adversarial perturbations into initial momentum for online iterative optimization for a single clean input image, and performs the following iterative optimization process: Extract the structural prior map from the clean input image, and use the structural prior map as a guiding mask to modulate the attack gradient to obtain the corrected attack gradient; In the frequency domain, simulated fine-tuning of the original features of the visual basic model causes feature drift, generating simulated bias features; A composite loss function is constructed based on the features of adversarial examples, the original features of the input clean image, and the simulated bias features. The attack gradient flow direction and the initial state are deeply reconstructed by the momentum iteration algorithm in combination with the composite loss function until the final adversarial example is generated.

[0058] It should be noted that the specific implementation of the domain adaptation adversarial attack system for visual basic models in this embodiment of the invention is similar to the specific implementation of the domain adaptation adversarial attack method for visual basic models in this embodiment of the invention. Please refer to the description in the method section for details. To reduce redundancy, it will not be repeated here.

[0059] Example 3 This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the domain adaptation adversarial attack method for vision-based models described above.

[0060] Example 4 This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the domain adaptation adversarial attack method for vision-based models described above.

[0061] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0062] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0063] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0064] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0065] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0066] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A domain-adaptive adversarial attack method for visual fundamental models, characterized in that, Includes the following steps: Offline domain alignment initialization phase: Construct a proxy dataset with a distribution similar to the target domain, and optimize the generation of general adversarial perturbations on the proxy dataset; Online instance-level adaptive phase: For a single clean input image, the general adversarial perturbation is transformed into initial momentum for online iterative optimization, and the following iterative optimization process is performed: Extract the structural prior map from the clean input image, and use the structural prior map as a guiding mask to modulate the attack gradient to obtain the corrected attack gradient; In the frequency domain, the feature drift caused by simulating fine-tuning of the original features of the visual basic model is used to generate simulated deviation features. A composite loss function is constructed based on the features of adversarial examples, the original features of the input clean image, and the simulated bias features. The attack gradient flow direction and the initial state are deeply reconstructed by the momentum iteration algorithm in combination with the composite loss function until the final adversarial example is generated.

2. The domain adaptation adversarial attack method for visual fundamental models as described in claim 1, characterized in that, The optimization objective of the general perturbation is to maximize the expected feature distortion of the base model encoder on the proxy dataset, which is mathematically expressed as: ， in, This represents the optimized general adversarial perturbation. Indicates in the proxy dataset Above is a clean input image Take the math period, An image encoder representing the fundamental model of vision. express Norm distance measures the degree of distortion in the feature space. This represents a candidate adversarial perturbation variable in the optimization process, whose value range is subject to a predefined perturbation bound. Constraints.

3. The domain adaptation adversarial attack method for visual fundamental models as described in claim 1, characterized in that, The calculation process of using structural priors as a guiding mask to modulate the attack gradient is as follows: ， in, No. t The attack gradient after the next iteration. It is the first t The original loss gradient calculated in the next iteration It is a scaling factor that controls the structural reinforcement strength. This indicates element-wise multiplication. Represents the structural prior diagram.

4. The domain adaptation adversarial attack method for visual fundamental models as described in claim 1, characterized in that, The process of simulating and fine-tuning the original features of the visual base model in the frequency domain to generate simulated bias features includes: The original features of the base model encoder are decomposed in the frequency domain using fast Fourier transform to obtain the phase spectrum and amplitude spectrum of the input clean image. Construct a spectral uncertainty profile that is proportional to the amplitude spectrum to capture potential feature drift; The random noise matrix following a standard Gaussian distribution is modulated using spectral uncertainty profiles, and the simulated bias characteristics are obtained through inverse fast Fourier transform.

5. The domain adaptation adversarial attack method for visual fundamental models as described in claim 1, characterized in that, In the parameter update during the offline domain alignment initialization phase, the sign-based momentum gradient descent method is used. In the offline iteration, the mathematical update formula for the general adversarial perturbation is: ， in, Indicates the first The general adversarial perturbation after offline iterative updates. Indicates the first General adversarial perturbation for offline iteration. The operator strictly constrains the obtained perturbation value to a pre-defined infinite norm boundary. Within the range, The set offline optimization step size, The function extracts the sign and direction of the gradient to ensure the stability of the update. For offline loss function, This represents the gradient operator of the offline loss function with respect to the perturbation variable.

6. The domain adaptation adversarial attack method for visual fundamental models as described in claim 1, characterized in that, The composite loss function is: ， in, Indicates the current number Features extracted from the visual base model based on the adversarial examples generated in each iteration. To input a clean image The original characteristics, In the first The feature drift is dynamically generated step-by-step to simulate the effect of unknown fine-tuning parameters. Indicates the first The instance-level adversarial perturbation is generated through step-by-step online iteration.

7. The domain adaptation adversarial attack method for visual fundamental models as described in claim 1, characterized in that, The combined composite loss function, through a momentum iteration algorithm, deeply reconstructs the attack gradient flow direction and the initial state, including: Using the domain-wide perturbation extracted in the offline phase, a hot-start initialization of the online momentum buffer is performed. In each iteration, the original gradient with respect to the input clean image is calculated for the composite loss function; The original gradient is spatially modulated using the structural prior graph to strengthen the edge attack weights, and the accumulated momentum is updated using the modulated gradient; the actual perturbation increment is calculated based on the updated accumulated momentum.

8. A domain-adaptive adversarial attack system for visual fundamental models, characterized in that, include: The offline domain alignment initialization module is used to construct a proxy dataset with a distribution similar to the target domain, and to optimize the generation of general adversarial perturbations on the proxy dataset. The online instance-level adaptive phase module is used to transform general adversarial perturbations into initial momentum for online iterative optimization for a single clean input image, and performs the following iterative optimization process: Extract the structural prior map from the clean input image, and use the structural prior map as a guiding mask to modulate the attack gradient to obtain the corrected attack gradient; In the frequency domain, the feature drift caused by simulating fine-tuning of the original features of the visual basic model is used to generate simulated deviation features. A composite loss function is constructed based on the features of adversarial examples, the original features of the input clean image, and the simulated bias features. The attack gradient flow direction and the initial state are deeply reconstructed by the momentum iteration algorithm in combination with the composite loss function until the final adversarial example is generated.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the domain adaptation adversarial attack method for vision-based fundamental models as described in any one of claims 1-7.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the domain adaptation adversarial attack method for vision-based fundamental models as described in any one of claims 1-7.

Images