SAM anti-attack method based on CLIP semantic guidance

Abstract

The invention relates to the field of confrontation attacks, in particular to an SAM confrontation attack method based on CLIP semantic guidance, which comprises a projector training stage and a confrontation attack stage, in the projector training stage, a CLIP semantic subspace is constructed based on positive and negative text prompts of segmentation quality, and meanwhile, an original image-damaged image pair is utilized to train a projector from SAM to CLIP, so that the confrontation attack is obtained. Through loss function optimization, embedding the projected image into the semantic subspace and aligning the projected image with the corresponding text semantics; in the anti-attack stage, the trained projector is frozen, and disturbance superposed on an original image is optimized by adopting a projection gradient descent method; the semantic guidance loss and the segmentation loss are jointly optimized to drive the embedding of the adversarial sample in the semantic subspace to be deviated to the negative text direction, and the segmentation output of the adversarial sample is enabled to be remarkably deviated from the original result; the method designed by the invention has higher semantic relevance and higher mobility.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of adversarial attack, and in particular to a SAM adversarial attack method based on CLIP semantic guidance. BACKGROUND

[0002] SAM (Segment Anything Model) is a model proposed by Meta Company to segment all models (SAM) as a milestone basic model for segmentation tasks. Adversarial attack is an important means to evaluate the security and robustness of the model, and its core goal is to add a small perturbation that is difficult for the human eye to detect to the original input to make the deep learning model produce false output. Through adversarial attack, the robustness boundary of the model in actual application can be evaluated, and technical support can be provided for the security reinforcement and privacy data protection of the model.

[0003] The existing SAM model adversarial attack method mainly has the following defects:

[0004] 1. Poor attack transferability. Since the SAM model structure is complex, it includes image encoders, prompt encoders, and mask decoders, etc. components, the adversarial perturbation generated by directly targeting a certain part of it will significantly decrease in effectiveness when attacking other variant SAM models or the same model under different configurations, i.e. poor transferability.

[0005] 2. Weak semantic correlation. Traditional attack methods mostly optimize in the pixel space or the feature space of the model, lacking high-level semantic guidance. The generated perturbation can destroy the segmentation result, but its destruction mode is random and unexplainable, and cannot achieve targeted and consistent attack guidance with human semantic understanding.

[0006] Therefore, there is an urgent need for a SAM adversarial attack method with attack transferability and semantic correlation. SUMMARY

[0007] Therefore, the present application discloses a SAM adversarial attack method based on CLIP semantic guidance to solve the above problems; comprising:

[0008] S1: Obtain an image data set and construct positive and negative text embedding vectors;

[0009] S2: Construct a pair of data sets containing clean images and destructive images, input the clean-damage image data set into the SAM image encoder with frozen parameters, and obtain the SAM features of the clean images and the destructive images;

[0010] S3: Input the SAM features of the clean images and the destructive images into the CLIP projector to obtain a pair of projected features consistent with the shallow feature dimension of the end multi-layer pyramid network of the CLIP image encoder;

[0011] S4: input the projection feature pair into the multi-layer pyramid network at the end of the CLIP image encoder with frozen parameters to obtain a SAM-to-CLIP embedding pair With ;

[0012] S5: transform the SAM-to-CLIP embedding pair and the positive and negative text embedding vectors into a segmentation semantic subspace to obtain a projection image embedding pair and a projection text embedding pair corresponding to the subspace;

[0013] S6: based on a cross-attention mechanism, couple the projection image embedding pair into the projection text embedding pair to obtain an updated positive and negative text embedding pair; during the coupling process, the projection image embedding serves as a query, and the projection text embedding serves as a key and a value;

[0014] S7: calculate a symmetric alignment loss and a direction preservation loss based on the projection image embedding pair and the updated positive and negative text embedding pair, and combine the losses to jointly optimize the SAM-to-CLIP projector;

[0015] S8: freeze the SAM-to-CLIP projector, and add an input perturbation to an original image to obtain an adversarial sample; input the original image and the adversarial sample into the SAM image encoder respectively to obtain an original image SAM feature and an adversarial sample SAM feature; the original image is different from a clean image;

[0016] S9: sequentially input the adversarial sample SAM feature into the SAM-to-CLIP projector and the multi-layer pyramid network of the CLIP image encoder and transform it into the segmentation semantic subspace to obtain a projected embedding vector;

[0017] S10: iteratively update the input perturbation according to the projected embedding vector pair, jump to S8, and continue until a preset number of iterations is reached to obtain a universal perturbation.

[0018] Based on the SAM-to-CLIP projector, the present application transfers part of the optimization process of the attack to the universal CLIP semantic space, so that the generated adversarial perturbation no longer depends heavily on the specific implementation of a specific SAM model, thereby effectively attacking SAM models and variants of different structures or configurations;

[0019] The present application utilizes the rich semantics contained in the CLIP model, and clearly defines the semantic direction of the attack through positive and negative text prompts, so that the generated perturbation is no longer blind noise, but an explainable pattern with clear semantic guidance, making the attack process more intelligent and controllable. The generated universal adversarial perturbation can be used to evaluate the robustness boundary of SAM-based models in practical applications on a large scale. It can also add an "immune protection layer" to images containing sensitive information to prevent automatic segmentation and identification by SAM and other models, thereby protecting personal privacy and business secrets. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 A schematic diagram of the SAM adversarial attack method based on CLIP semantic guidance in the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical scheme, characteristics and advantages of the present application more clear and obvious, and to make the technical personnel in the art better understand the technical scheme of the present application, the present application will be further described in detail below in combination with the drawings and examples.

[0022] The present embodiment includes a SAM adversarial attack method based on CLIP semantic guidance, wherein the SAM at least comprises: an image encoder, inputting a picture and outputting an image feature; a prompt encoder, inputting a prompt (point, box) and outputting a corresponding feature; and a decoder, inputting an image feature and a prompt feature and outputting a segmentation result.

[0023] The adversarial attack method comprises:

[0024] S1: Obtain an image data set and construct positive and negative text embedding vectors; the image data set comprises clean images and original images.

[0025] Specifically, clean images are extracted from the data set (including SA1B, ADE20K, Cityscapes and MS COCO). The clean images are used to train the projector, and the original images are used for subsequent iterative updating of the perturbation.

[0026] The construction of positive and negative text embedding vectors specifically involves building a set of text pairs containing positive and negative text prompts related to segmentation features. In this embodiment, this set is constructed based on the large language model GPT-3.5. For example, positive text: "the image shows sharp and well-defined object edges.", negative text: "the image shows blurry and poorly-defined object edges.", generating 22 positive and 22 negative text pairs respectively. The positive text embedding vector is obtained through a frozen pre-trained CLIP text encoder. and negative text embedding vectors .

[0027] S2: Construct a pairwise dataset containing clean and destructive images. Input the clean-destructive image dataset into the SAM image encoder with frozen parameters to obtain the SAM features of the clean and destructive images. The destructive images are obtained by destroying the clean images. The destruction methods include: blurring, pixelation, random occlusion, and color transformation.

[0028] S3: Input the SAM features of the clean image and the destructive image into the CLIP projector to obtain a projection feature pair that is consistent with the shallow feature dimension of the multi-layer pyramid network at the end of the CLIP image encoder.

[0029] In this embodiment, the CLIP image encoder uses the RN50x16 version based on a convolutional neural network. The SAM features of the clean and destructive images are input into the CLIP projector. In the process, a multi-layer pyramid network at the end of the CLIP image encoder is obtained. Shallow feature dimension consistent projection feature pairs and .

[0030] S4: Input the projected feature pairs into a multi-layer pyramid network at the end of the CLIP image encoder with frozen parameters to obtain SAM-to-CLIP embedding pairs. and .

[0031] S5: Transform the SAM to CLIP embedding pairs and positive and negative text embedding vectors to the segmented semantic subspace to obtain the projected image embedding pairs and projected text embedding pairs for the corresponding subspace. Specifically:

[0032] S51: Based on the embedding vector of the main text and negative text embedding vectors Construct the difference vector The formula is:

[0033]

[0034] S52: Construct the semantic subspace of the segmentation feature in the CLIP, perform singular value decomposition (SVD) on the difference vector , and extract the principal components as the segmentation semantic subspace basis ; the formula is:

[0035]

[0036]

[0037] wherein, represents the left singular vector matrix, represents the singular value diagonal matrix, represents the right singular vector matrix; The first right singular vectors of form the segmentation semantic subspace basis , and in the present embodiment 10 is taken.

[0038] S53: Transform the SAM-to-CLIP embedding pair and the positive and negative text embedding vectors into the segmentation semantic subspace; the formula is:

[0039]

[0040]

[0041]

[0042]

[0043] wherein, and respectively represent the projection image embedding pair corresponding to the clean image and the attack image, and respectively represent the projection text embedding pair corresponding to the positive text prompt and the negative text prompt

[0044] S6: Based on the cross-attention mechanism, couple the projection image embedding pair into the projection text embedding pair to obtain the updated positive and negative text embedding pair; in the coupling process, the projection image embedding serves as the query, and the projection text embedding serves as the key and the value; the formula is:

[0045]

[0046]

[0047] wherein, and denote the updated positive and negative text embedding pairs, denote each Token dimension, 768 in this embodiment.

[0048] S7: According to the projection image embedding pair and the updated positive and negative text embedding pair, calculate the symmetric alignment loss and the direction preservation loss , jointly optimize the SAM to CLIP projector; the formula is:

[0049]

[0050]

[0051]

[0052]

[0053]

[0054] wherein B denotes the batch size, and are the image projection difference vector and the text projection difference vector respectively, denote the change direction thereof in the feature space, and denote the loss coefficients, in this embodiment and are respectively set to 1 and 0.5.

[0055] The alignment loss ensures that the projected SAM features and the corresponding CLIP text embedding are closely aligned in the semantic space, while the direction alignment loss constrains the change direction of the visual features from the corrupted sample to the clean sample to be consistent with the change direction of the text semantics from negative to positive, thereby realizing the optimization of the SAM to CLIP projector.

[0056] S8: Freeze the SAM to CLIP projector, add input perturbations to the original image to obtain the adversarial sample ; input the original image and the adversarial sample into the SAM image encoder respectively to obtain the original image SAM feature and the adversarial sample SAM feature.

[0057] The original image is selected from the data set, and the original image is different from the clean image used in the process of training the projector in S1-S7. In this embodiment, 500 original images are selected, and each original image undergoes S8-S12 to constantly update the input perturbation .

[0058] S9: The adversarial sample SAM features are sequentially input into the SAM to the CLIP projector and the CLIP image encoder multi-layer pyramid network, transformed to the segmentation semantic subspace, and the projected embedding vector is obtained ;

[0059] S10: The projected embedding vector is combined with the projection positive and negative text embedding pair 、 to calculate the semantic guidance loss , and the formula is:

[0060]

[0061]

[0062]

[0063] S11: The original image SAM features and the adversarial sample SAM features and the corresponding segmentation prompt features are input into the SAM decoder to obtain the corresponding logical segmentation output and , and the segmentation loss is calculated according to the logical segmentation output.

[0064]

[0065] S12: The attack loss is constructed by combining the semantic guidance loss and the segmentation loss , and the input perturbation is iteratively updated through the projection gradient descent algorithm (PGD); the formula is:

[0066]

[0067]

[0068] wherein, and respectively represent the weights of the semantic guidance loss and the segmentation loss, which are respectively set to 0.4 and 0.6 in the embodiment, represents a clipping function, which is used to ensure that the amplitude range of the perturbation is within , and is set to in the embodiment, represents the number of iterations, represents the step size, represents a sign function.

[0069] S13: Steps S8-S12 are repeated until a preset number of iterations is reached.

[0070] In this embodiment, the number of iterations is set to 500 rounds, and the general perturbation is finally obtained.

[0071] Further, as Figure 1 shown, in the training stage, in terms of text, the obtained positive and negative prompt texts are input into the frozen CLIP text encoder to obtain the corresponding text embeddings and , and the difference is obtained by SVD to obtain the subspace basis. In terms of images, the clean images in the dataset are degraded to obtain clean-damaged image pairs, which are input into the frozen SAM image encoder to obtain the corresponding SAM features, and then the above SAM features are sequentially input into the SAM2CLIP projector and the CLIP multi-layer pyramid to obtain the SAM-to-CLIP embedding pair and . Then the obtained text embeddings and SAM-to-CLIP embedding pairs are respectively obtained by the subspace basis to obtain the projection embedding, and then the projected image-text embedding is fused by cross-attention to obtain the updated projection text embedding, and then the projection text embedding is used to optimize the SAM2CLIP projector.

[0072] In the test stage, the clean image is added to the initialization perturbation to obtain the adversarial sample, the clean image and the adversarial sample are input into the frozen SAM image encoder to obtain the SAM feature, and the segmentation prompt feature is input into the SAM decoder to obtain the corresponding segmentation output, and the segmentation loss is calculated. At the same time, the SAM feature of the above obtained adversarial sample is sequentially input into the frozen SAM2CLIP projector and the frozen CLIP multi-layer pyramid network, and the result is projected into the segmentation semantic subspace through the subspace basis, and then the semantic guidance loss is calculated by combining the projection positive and negative text embeddings.

[0073] Finally, it should be noted that the above only describes part of the embodiments of the present application, and those skilled in the art can make various changes, modifications, replacements and deformations to these embodiments without departing from the principles and spirits of the present application, the protection scope of the present application is defined by the appended claims and their equivalents, and the above behaviors should be covered within the protection scope of the present application.

Claims

1. A SAM adversarial attack method based on CLIP semantic guidance, characterized in that, include: S1: Obtain the image dataset and construct positive and negative text embedding vectors; the image dataset includes clean images and original images; S2: Construct a pairwise dataset containing clean and destructive images. Input the clean-destructive image dataset into the SAM image encoder with frozen parameters to obtain the SAM features of the clean and destructive images. S3: Input the SAM features of the clean image and the destructive image into the CLIP projector to obtain a projection feature pair that is consistent with the shallow feature dimension of the multilayer pyramid network at the end of the CLIP image encoder; S4: Input the projected feature pairs into a multi-layer pyramid network at the end of the CLIP image encoder with frozen parameters to obtain SAM-to-CLIP embedding pairs. and ; S5: Transform the SAM to CLIP embedding pairs and positive and negative text embedding vectors to the segmented semantic subspace to obtain the projected image embedding pairs and projected text embedding pairs in the corresponding subspace; S6: Based on the cross-attention mechanism, the projected image embedding pair is coupled to the projected text embedding pair to obtain the updated positive and negative text embedding pair; During the coupling process, the projected image embedding serves as the query, and the projected text embedding serves as the key and value; S7: Calculate the symmetry alignment loss based on the projected image embedding pairs and the updated positive and negative text embedding pairs. Loss with direction Combined with loss analysis, the SAM to CLIP projector is optimized. S8: Freeze SAM to CLIP projector and add input perturbation to the original image. Adversarial examples are obtained; the original image and the adversarial examples are respectively input into the SAM image encoder to obtain the SAM features of the original image and the SAM features of the adversarial examples; the original image is distinguished from the clean image. S9: Input the adversarial sample SAM features sequentially into the SAM, then into the CLIP projector and CLIP image encoder multi-layer pyramid network and transform them into the segmentation semantic subspace to obtain the projected embedding vector. S10: Iteratively update the input perturbation based on the projected embedding vector, jump to S8, until the preset number of iterations is reached to obtain the general perturbation.

2. The SAM adversarial attack method based on CLIP semantic guidance according to claim 1, characterized in that, The destructive image is obtained by destroying a clean image. The destruction methods include: blurring, pixelation, random occlusion, and color transformation.

3. The SAM adversarial attack method based on CLIP semantic guidance according to claim 1, characterized in that, The process of obtaining the projected image embedding pair and the projected text embedding pair for the corresponding subspace includes: S51: Based on the embedding vector of the main text and negative text embedding vector Construct the difference vector; S52: Construct a semantic subspace for segmentation features in CLIP, perform singular value decomposition on the difference vector, and extract principal components as the basis of the segmentation semantic subspace; S53: Transform the SAM to CLIP embedding pairs and positive and negative text embedding vectors into the segmented semantic subspace.

4. The SAM adversarial attack method based on CLIP semantic guidance according to claim 3, characterized in that, The formula for transforming the SAM-to-CLIP embedding pairs and positive and negative text embedding vectors to the segmented semantic subspace is as follows: in, and These represent the projected image embedding pairs corresponding to the clean image and the aggressive image, respectively. and These represent the projected text embedding pairs corresponding to the positive and negative text hints, respectively. It represents the basis of the segmented semantic subspace.

5. The SAM adversarial attack method based on CLIP semantic guidance according to claim 1, characterized in that, The formula for coupling the projected image embedding pair to the projected text embedding pair is as follows: in, and This represents the updated positive and negative text embedding pairs. and These represent the projected image embedding pairs corresponding to the clean image and the aggressive image, respectively. and These represent the projected text embedding pairs corresponding to the positive and negative text hints, respectively. This represents each token dimension.

6. The SAM adversarial attack method based on CLIP semantic guidance according to claim 1, characterized in that, The iterative update of the input perturbation based on the projected embedding vector includes: Step 1: Calculate the semantic guidance loss based on the projected embedding vector and the projected positive and negative text embedding pairs; Step 2: Input the original image SAM features, adversarial sample SAM features, and corresponding segmentation cue features into the SAM decoder to obtain the corresponding logical segmentation output, and calculate the segmentation loss based on the logical segmentation output; Step 3: Construct the attack loss by combining the semantic guidance loss and the segmentation loss, and iteratively update the input perturbation using the projection gradient descent algorithm.

7. The SAM adversarial attack method based on CLIP semantic guidance according to claim 6, characterized in that, The semantic guidance loss is calculated as follows: in, Indicates embedded text The nth element, Indicates negative text embedding The nth element, Indicates semantic guidance loss. This represents the embedded vector after projection.

8. The SAM adversarial attack method based on CLIP semantic guidance according to claim 6, characterized in that, The iterative update of the input perturbation is performed using the following formula: in, Indicates damage caused by the attack. and These represent the weights of the semantic guidance loss and the segmentation loss, respectively. This represents the pruning function used to ensure perturbation The range of amplitude is within, Indicates the number of iterations. Indicates the step size. Represents a symbolic function.

Images