A method and device for multi-target adversarial attack based on text description

Abstract

Embodiments of the present application provide a text description-based multi-target adversarial attack method and device. In the embodiments of the present application, a text feature in a target text is extracted based on a text encoder in a preset model; an image feature in an original image is extracted based on an image encoder in the preset model; the text feature and the image feature are fused based on a semantic fusion module in the preset model to obtain a fused semantic feature; an adversarial perturbation image is generated based on a generator in the preset model; and the adversarial perturbation data image is added to the original image to generate a corresponding adversarial image sample. Thus, the generation efficiency and flexibility of the adversarial sample are improved.

Description

Technical Field

[0001] This invention relates to the field of image processing, specifically to a method and apparatus for multi-target adversarial attacks based on text description. Background Technology

[0002] With the widespread application of deep neural networks (DNNs) in fields such as image classification and speech recognition, their security has received increasing attention. Research has found that DNNs are significantly vulnerable to adversarial examples, which can mislead the model's decisions by adding perturbations that are difficult for humans to detect.

[0003] In related technologies, generating adversarial examples using Generative Adversarial Networks (GANs) can significantly improve the efficiency of adversarial attacks. However, this approach is usually limited to a single target class, requiring retraining the generative model for each target class, which increases computational costs. This is particularly problematic in scenarios where attacks need to target multiple target classes, resulting in low efficiency and insufficient flexibility. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to overcome the shortcomings of the existing technology and provide a multi-target adversarial attack method based on text description, which improves attack efficiency and flexibility by targeting adversarial perturbations for multiple target categories.

[0005] To achieve the aforementioned technical objectives, this invention provides a text-based multi-target adversarial attack method, comprising: extracting text features from target text using a text encoder in a pre-set model, wherein the target text has a corresponding target category; extracting image features from an original image using an image encoder in the pre-set model, wherein the category of the original image is inconsistent with the target category; fusing the text features and the image features using a semantic fusion module in the pre-set model to obtain fused semantic features; generating an adversarial perturbation image using the semantic features using a generator in the pre-set model; and adding the adversarial perturbation image to the original image to generate corresponding adversarial image samples.

[0006] Specifically, the semantic fusion module in the pre-set model fuses the text features and the image features to obtain fused semantic features, including: using a linear transformation in the semantic fusion model to embed the text features as a query, and using the latent encoding of the text features as keys and values ​​to determine an attention map in the spatial and channel dimensions; based on the attention map, determining the parameters of the linear transformation, and using the parameters to map the text features to the latent subspace of the corresponding pre-set model, wherein the latent subspace is a subspace of the image features, so that the text features and image features are fused.

[0007] Specifically, the step of generating an adversarial perturbation image based on the generator in the preset model includes: based on the text feature distribution of the target category, the generator generates an adversarial perturbation image from the semantic features, so as to shift the semantic features of the adversarial image sample to the feature center of the target category through the adversarial perturbation image.

[0008] Specifically, generating an adversarial perturbation image from the semantic features based on the generator in the preset model includes: decoding the semantic features based on the decoder in the generator to obtain the adversarial perturbation image.

[0009] Furthermore, the method also includes: generating a composite loss function based on the cross-entropy loss function, the KL divergence loss function, and a regularization term; and generating the generator based on the composite loss function.

[0010] The pre-built model includes the contrastive language-image pre-training CLIP framework.

[0011] In addition, the method also includes: using the encoder in the generator to extract image features from the original image; after obtaining the fused semantic features, concatenating the semantic features with the extracted image features; and decoding the concatenated image features through the decoder in the generator to obtain the adversarial image.

[0012] Furthermore, the method also includes: receiving an image of a first preset size for carrying the text features and an image of a first preset size for carrying the image features through a multi-scale fusion module in the preset model, and generating a text feature image and three feature images of different scales corresponding to the image feature image through adaptive pooling operations; copying the three feature images of different scales along the spatial dimension to generate a feature image of the first preset size for each of the text feature image and the image feature image; receiving the original image and using a lightweight network to extract features from the original image to determine the fusion weights of the multi-scale features, which are applied to feature maps of different scales; and performing weighted fusion on the three feature images of different scales and the corresponding feature images of the first preset size based on the fusion weights to obtain a feature image of the second preset size for fusing text features and image features.

[0013] In addition, the method further includes: after generating the adversarial image sample, obtaining the high-frequency components of the image sample, and constraining the high-frequency components.

[0014] On the other hand, the present invention provides a multi-target adversarial attack device based on text description, comprising: a first extraction module, used to extract text features from target text based on a text encoder in a preset model, wherein the target text has a corresponding target category; a second extraction module, used to extract image features from an original image based on an image encoder in the preset model, wherein the category of the original image is inconsistent with the target category; a fusion module, used to fuse the text features and the image features based on a semantic fusion module in the preset model to obtain fused semantic features; a generation module, used to generate an adversarial perturbation image based on the semantic features using a generator in the preset model; and an adding module, used to add the adversarial perturbation image to the original image to generate corresponding adversarial image samples.

[0015] In this embodiment, text features are extracted from the target text based on the text encoder in the preset model, and the target text has a corresponding target category; image features are extracted from the original image based on the image encoder in the preset model, and the category of the original image is inconsistent with the target category; the text features and the image features are fused based on the semantic fusion module in the preset model to obtain fused semantic features; an adversarial perturbation image is generated based on the semantic features using the generator in the preset model; and the adversarial perturbation image is added to the original image to generate corresponding adversarial image samples.

[0016] This method achieves multimodal feature extraction by extracting text features from the target text using a pre-built text encoder and image features from the original image using a pre-built image encoder. The text and image features are then fused using a semantic fusion module within the pre-built model. Finally, a generator within the pre-built model generates corresponding adversarial examples. This allows for the dynamic generation of different target or multi-target adversarial perturbation images based on varying target requirements, which can then be applied to the input image to generate adversarial examples. No retraining of the generator is required, thus improving the efficiency and flexibility of adversarial example generation. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating a text-based multi-target adversarial attack method according to an embodiment of this application.

[0019] Figure 2 This is a schematic diagram illustrating a text-based multi-target adversarial attack process according to an embodiment of this application;

[0020] Figure 3A This is a schematic diagram illustrating the relationship between the number of categories and the attack success rate in an embodiment of this application.

[0021] Figure 3B This is a schematic diagram illustrating the relationship between the number of categories and the attack success rate in an embodiment of this application.

[0022] Figure 4 This is a schematic diagram illustrating a specific example of an embodiment of this application;

[0023] Figure 5 This is a schematic diagram of the framework of a text-based multi-target adversarial attack device according to an embodiment of this application;

[0024] Figure 6 This is a schematic diagram of the framework of a text-based multi-target adversarial attack device according to an embodiment of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] like Figure 1 As shown in the embodiments of this application, a multi-target adversarial attack method based on text description is also provided. The method 100 includes:

[0027] S101: Extract text features from the target text based on the text encoder in the pre-built model.

[0028] The target text has a corresponding target category.

[0029] S102: Extract image features from the original image based on the image encoder in the preset model.

[0030] In this case, the category of the original image is inconsistent with the category of the target image.

[0031] S103: Based on the semantic fusion module in the pre-built model, text features and image features are fused to obtain fused semantic features.

[0032] S104: Generate adversarial perturbation images based on the generator in the pre-built model by using semantic features.

[0033] S105: Add the adversarial perturbation image to the original image to generate corresponding adversarial image samples.

[0034] It should be noted that this method 100 can be applied to server-side, service cluster-side, or cloud-based server configurations.

[0035] The following is a detailed explanation of the above steps:

[0036] S101: Extract text features from the target text based on the text encoder in the pre-built model.

[0037] The target text has a corresponding target category. For example, such as Figure 2 As shown, the target text can be "monitor", and its category can be "monitor".

[0038] The aforementioned preset model can be a known model, any self-made model, or a variant of a known model. However, regardless of the model, it should have two or more modal encoders, such as a text encoder and an image encoder, and other modal encoders can be added as needed.

[0039] The pre-built model includes the contrastive language-image pre-trained CLIP framework. This framework can include encoders such as text encoders and image encoders.

[0040] For example, as described above, and as... Figure 2 As shown, a text description (monitor) with a target category can be input into the CLIP text encoder to extract text features of the target category. The extracted text features are represented as high-dimensional embedding vectors of the target category, used to describe the semantic information of the target category.

[0041] S102: Extract image features from the original image based on the image encoder in the preset model.

[0042] In this case, the category of the original image is inconsistent with the target category. The original image could be a vehicle image, in which case the category should be "vehicle," i.e., "vehicle image." However, the text category (e.g., "monitor") does not correspond to the vehicle image; that is, the target category of the text is used to distract from the vehicle image. The goal is to ensure that the generated adversarial example is misclassified as the target category by the subsequent target classifier.

[0043] As stated above, and as Figure 2 As shown, the original input image can also be fed into the CLIP image encoder (i.e., image encoder) to extract the image's visual features, i.e., image features. Image features are represented as multidimensional embedding vectors of the original input image, used to describe the visual content.

[0044] Since this embodiment can process text and image modalities simultaneously, it extracts and fuses features from the input image and the text description of the target category, and dynamically generates adversarial perturbations adapted to the target category through a generative model. Thus, through CLIP's multimodal capabilities, it can dynamically combine text descriptions and image features to generate multi-target adversarial perturbations, and apply them to the input image to generate adversarial examples.

[0045] S103: Based on the semantic fusion module in the pre-built model, text features and image features are fused to obtain fused semantic features.

[0046] The semantic fusion module establishes a mapping relationship between the image's feature space and CLIP's text embedding space (i.e., the text feature space). Through this mapping, the model can generate image perturbations based on various semantic textual cues, thereby enabling adversarial attacks.

[0047] As mentioned above, Figure 2 As shown, the extracted text features and image features are input into the semantic module (i.e., the semantic fusion module) for fusion.

[0048] Specifically, the semantic module may include a semantic modulation module and a fusion module. The semantic modulation module receives the aforementioned features, and during the fusion process, text features and image features interact through the semantic modulation mechanism within this module to form a unified feature representation.

[0049] The semantic modulation module employs linear mapping and a bidirectional attention mechanism, calculating attention maps in both the spatial and channel dimensions to refine the injection of semantic information. The fusion module then generates fused features that not only incorporate the visual features of the input image but also integrate textual semantic information of the target category, laying the foundation for subsequent adversarial perturbation generation.

[0050] Specifically, based on the semantic fusion module in the pre-built model, text features and image features are fused to obtain fused semantic features. This includes: using the linear transformation in the semantic fusion model to embed text features as queries and using the latent encoding of text features as keys and values ​​to determine attention maps in the spatial and channel dimensions; based on the attention maps, determining the parameters of the linear transformation and using the parameters to map text features to the corresponding latent subspace of the pre-built model, where the latent subspace is a subspace of image features, so as to fuse text features with image features.

[0051] The semantic fusion module employs a linear transformation, using text embeddings (text features) as queries and latent encodings as keys and values, and calculates their cross-attention. This process is performed separately in the spatial and channel dimensions, resulting in two attention maps. Based on these attention maps, the parameters of the linear transformation (e.g., translation and scaling parameters) are calculated. These parameters are then used to map the text cue embeddings (text features) to the corresponding CLIP latent subspace, i.e., the space of image features, for feature fusion.

[0052] This module employs a two-dimensional attention mechanism (spatial and channel-based) for semantic fusion, learning how to align from visual and textual details. For example, in the spatial dimension, scaling parameters are used to adjust the contribution of each spatial location, while in the channel dimension, translation parameters for each channel are fine-tuned to precisely adjust the matching with the latent space of the aforementioned image features.

[0053] It should be understood that the semantic fusion module here can also be divided into a semantic modulation module and a fusion module.

[0054] S104: Generate adversarial perturbation images based on the generator in the pre-built model by using semantic features.

[0055] As described above, the fused semantic features are input into the generator to generate adversarial perturbation images, which are then added to the original input image to form adversarial examples.

[0056] Among them, such as Figure 2 As shown, the generator can include the aforementioned semantic module (i.e., semantic fusion module), or the semantic module can be decoupled from the generator. When the generator includes the aforementioned semantic module, the generator can directly generate adversarial perturbation images based on the semantic features, i.e., as shown... Figure 2 The counter-disturbance p shown x，t The counter-perturbation data can be image data.

[0057] Specifically, based on the generator in the pre-built model, adversarial perturbation images are generated from semantic features, including: based on the decoder in the generator, the semantic features are decoded to obtain adversarial perturbation images.

[0058] As mentioned above, in the process of generating adversarial perturbations, a semantic feature decoder, namely a generator decoder, is used to decode the semantic features to obtain the corresponding perturbations.

[0059] To generate perturbations more accurately, the generator needs to generate perturbations according to the following mechanism.

[0060] Specifically, based on the generator in the pre-built model, an adversarial perturbation image is generated from semantic features, including: based on the text feature distribution of the target category, the generator generates an adversarial perturbation image from the semantic features, so as to shift the semantic features of the adversarial image sample to the feature center of the target category through the adversarial perturbation image.

[0061] As mentioned above, the generator calculates the feature offset of the input original image as the difference between the target category feature and the current category feature, thereby optimizing the generated adversarial examples so that they can be more accurately misclassified as the target category by the target classifier.

[0062] The target category is the target category of the text mentioned above, which serves as the interference category, while the current category is the category to which the original image belongs.

[0063] The generator can include two modules: a feature input module and a perturbation generation module. The feature input module receives fused image and text features, while the perturbation generation module generates adversarial perturbations through multi-layer convolution operations. The decoder mentioned above can be set within the perturbation generation module to perform deconvolution operations.

[0064] To generate perturbations that better match the specific features of the original image input, image stitching of image features and semantic features can also be performed on the original image to generate perturbations.

[0065] Specifically, the method 100 further includes: extracting image features from the original image using the encoder in the generator; concatenating the semantic features with the extracted image features after obtaining the fused semantic features; and decoding the concatenated image features using the decoder in the generator to obtain an image against perturbation.

[0066] like Figure 2 As shown above, the original image is input into the generator encoder in the generator to perform a convolution operation to obtain the image features of the original image. The image features and semantic features are then concatenated to obtain the concatenated image features. Finally, the generator decoder performs a deconvolution operation to obtain the image against perturbation.

[0067] It should be noted that the aforementioned, such as Figure 2 The image encoder shown extracts a 512-dimensional vector of image features, while the generator encoder is a convolutional network used to extract spatial features. Furthermore, as mentioned earlier, it is also possible to bypass the generator encoder on the original image and directly decode it using semantic features to obtain an adversarial image.

[0068] Furthermore, the generator is obtained through training, but the generator in this embodiment does not require multiple training sessions to meet the needs of multiple categories.

[0069] Specifically, the method 100 also includes: generating a composite loss function based on the cross-entropy loss function, the KL divergence loss function, and the regularization term; and generating a generator based on the composite loss function.

[0070] To enable multi-target adversarial attacks, the generator uses a composite loss function to optimize the generation process. This loss function consists of the following three parts:

[0071] Cross-entropy loss function: ensures that the generated adversarial examples are misclassified as the target category by the subsequent target classifier. The specific formula is as follows:

[0072]

[0073] in, Predict adversarial examples for the model Category The probability distribution is given by q, where C is the total number of categories, i.e., the number of possible categories in the classification task. c For target category One-hot encoding for class c.

[0074] The KL divergence loss function optimizes the similarity between the distribution of adversarial examples and the features of the target class. The specific formula is as follows:

[0075]

[0076] in, The function is used to calculate the similarity between two features. Image features for adversarial examples, For the text features of the target category, KL is the KL divergence calculation function. This is for model prediction of adversarial examples. Category The probability distribution.

[0077] Regularization term: Constrains the magnitude of adversarial perturbations, ensuring the imperceptibility of adversarial examples. The specific formula is as follows:

[0078]

[0079] Where x is the original image.

[0080] Combining the above three parts, the final composite loss function is:

[0081]

[0082] Here, α, β, and γ are hyperparameters that adjust the weights of each loss term.

[0083] S105: Add the adversarial perturbation image to the original image to generate corresponding adversarial image samples.

[0084] Among them, the adversarial image samples, namely the adversarial samples mentioned above, are used for training the subsequent target classifier.

[0085] As described above, the generated adversarial perturbation is added to the original image to obtain adversarial examples. The adversarial examples satisfy the following constraint: the perturbation amplitude is limited to... Within the threshold range, the perturbation is ensured to be imperceptible to the human eye while effectively misleading the target classifier. The resulting adversarial examples can target multiple target categories simultaneously, enabling dynamic switching between multi-target adversarial attacks.

[0086] like Figure 2 As shown, in generating adversarial examples x itj adv Next, the adversarial examples are fed into a white-box model (target classification model or target classifier) ​​to obtain the predicted class probability distribution of the adversarial examples. The first part of the training loss is obtained by calculating the cross-entropy loss. Subsequently, the second part of the training loss is obtained by calculating the probability distribution of the similarity between the target class text features and the text features of all classes and the class probability distribution predicted by the white-box model. The white-box model is then updated and iterated using the two parts of the training loss.

[0087] like Figure 4 As shown, the final generated adversarial image samples include vehicle images with the text category "rooster", traffic segment images with the text category "kangaroo", and traffic signs with the text category "tiger".

[0088] To verify the effectiveness of the proposed method in this embodiment, experiments were conducted on multiple datasets, and an ablation study was performed to investigate the impact of different modules and loss functions. Experimental results show that the proposed method has significant performance advantages in multi-target adversarial attack tasks.

[0089] The experiments used the following publicly available datasets: CIFAR10: Contains 60,000 32×32 color images, divided into 10 classes, with 6,000 images per class. CIFAR100: Extended from CIFAR10, it contains 100 classes, with 600 images per class. ImageNet-1k: Contains 1,000 classes, with at least 1,000 images per class, offering broad coverage and rich category diversity.

[0090] In addition, such as Figure 3AAs shown in the figure, regarding the relationship between attack success rate and the number of categories in the CIFAR10 dataset, three different lines can be seen in the graph. These three lines represent the attack success rate of the category with the highest attack success rate, the average attack success rate of the categories, and the attack success rate of the category with the lowest attack success rate, respectively. From high to low, the attack success rate decreases with the number of categories. Figure 3B As shown in the figure, regarding the relationship between attack success rate and the number of categories in the CIFAR100 dataset, three different lines can be seen. These three lines represent the attack success rate of the category with the highest success rate, the average attack success rate of each category, and the attack success rate of the category with the lowest success rate, respectively. The attack success rate decreases from high to low with the number of categories. Therefore, it can be concluded that the attack success rate decreases as the number of categories increases. The embodiments of this application implemented the following experiments to improve the attack success rate.

[0091] The experiments were conducted on a single NVIDIA 2080 GPU, with training lasting approximately 10 hours. The inference time for generating adversarial perturbations per image was 0.015 seconds. The CLIP framework used in the experiments was a pre-trained ViT-B / 32, with the weight parameters in the loss function set to α=250, β=250, and γ=1. Training lasted for a total of 120 epochs, with an initial learning rate of... It decreased to after 60 epochs. For different datasets, the perturbation amplitude is limited. The values ​​were set to 16 (CIFAR10), 32 (CIFAR100), and 64 (ImageNet-1k), respectively.

[0092] In the CIFAR10 and CIFAR100 datasets, the top 10 categories were selected as target attack categories. For the ImageNet-1k dataset, since the top 10 categories are semantically similar, 10 target attack categories were randomly selected from this dataset. All experiments were conducted for training and testing on the datasets.

[0093] To evaluate the effectiveness of the proposed method, ablation experiments were conducted on the CIFAR10 and CIFAR100 datasets. First, the impact of the semantic module in the generator and the KL divergence term in the loss function was evaluated. Attack success rate (ASR) was used as the evaluation metric to measure attack performance. The experimental results are shown in Table 1 below. The results show that introducing the semantic module improved the average attack success rate for all ten categories on both the CIFAR10 and CIFAR100 datasets, indicating that the module effectively embeds semantic information into the image modality. Adding the KL divergence term to the loss function also slightly improved the attack success rate. Notably, the best experimental results were obtained when both the semantic module and the KL divergence term were used simultaneously.

[0094] Overall, the embodiments of this application significantly improve the effectiveness of adversarial attacks. The introduction of the semantic module not only enhances the semantic consistency of the generated adversarial examples but also provides clearer guidance for their generation. Simultaneously, the introduction of the KL divergence term further optimizes the generator's loss function structure, resulting in adversarial examples that better match the characteristics of the target model.

[0095] Table 1

[0096]

[0097] ClipAdv refers to the CLIP framework mentioned above, and sem refers to the semantic module mentioned earlier.

[0098] The method proposed in this embodiment is applicable to various datasets (such as CIFAR10 and ImageNet-1k), and its performance and effectiveness can be verified in tasks such as classification and detection. Experiments show that this embodiment can achieve efficient adversarial attacks on multiple target categories simultaneously, demonstrating strong versatility and robustness.

[0099] In order to generate adversarial perturbations at different scales to better attack images, this embodiment also incorporates a multi-scale fusion module. The multi-scale fusion module aims to effectively integrate feature information at different scales to improve feature representation and robustness.

[0100] Specifically, the method 100 further includes: receiving an image of a first preset size for carrying text features and an image of a first preset size for carrying image features through a multi-scale fusion module in a preset model, and generating a text feature image and three feature images of different scales corresponding to the image feature image through adaptive pooling operations; copying the three feature images of different scales along the spatial dimension to generate a feature image of the first preset size for the text feature image and the image feature image respectively; receiving the original image and using a lightweight network to extract features from the original image to determine the fusion weights of the multi-scale features, which are applied to feature maps of different scales; and performing weighted fusion on the three feature images of different scales and the corresponding feature images of the first preset size based on the fusion weights to obtain a feature image of the second preset size for fusing text features and image features.

[0101] It should be noted that this multi-scale fusion module can be set before the semantic fusion module.

[0102] For example, as described above, this module first receives text features and image features from the CLIP encoder, with the corresponding text features and image features belonging to an input feature map of size C×H×W. It then generates three feature maps of different scales through adaptive pooling: C×H / 2×W / 2, C×H / 4×W / 4, and C×H / 8×W / 8. Subsequently, it restores the original size using a copy-and-paste method, that is, copying the three scaled feature maps along the spatial dimension to form a single C×H×W feature map—a text feature map and an image feature map of that size. In addition, this module receives an additional original image as described above and uses a lightweight network (e.g., a convolutional network) to extract features from it, learning the fusion weights of the aforementioned multi-scale features. These weights are applied to the four feature maps of different scales (C×H / 2×W / 2, C×H / 4×W / 4, C×H / 8×W / 8, and C×H×W) to control their contribution in the fusion process. Finally, after weighted fusion, a feature map of size 4C×H×W (such as image features and text features) is obtained, which contains information from different scales. It also features adaptive weighting based on image content, improving the model's ability to represent multi-scale information. The weighted fused image features can then be input into the semantic fusion module for further fusion.

[0103] To better inject semantic information into adversarial perturbations, this embodiment also incorporates Adaptive Instance Normalization (AdaIN). AdaIN adjusts the mean and standard deviation by introducing external semantic features s, thereby fusing semantic information during the normalization process. Specifically, given content features x and external semantic features s, AdaIN calculates:

[0104]

[0105] in and These are the mean and standard deviation of the content features. and These are new mean and standard deviations calculated from external semantic features.

[0106] Since s is a global semantic vector, the example uses a multilayer perceptron (MLP) to calculate the normalization parameters corresponding to s:

[0107]

[0108] in Used to scale feature distribution, It is used to adjust the mean of features. MLP is a simple two-layer fully connected network.

[0109] Finally, the semantic features s are obtained through MLP computation. and This is then applied to the AdaIN normalization formula to dynamically adjust the content feature x. Here, the content feature x refers to the fused semantic feature, while the semantic feature s is the newly introduced semantic feature from the outside.

[0110] Generating perturbations with high transferability is a significant challenge in adversarial attacks. To improve the universality of perturbations across different models and inputs, a high-frequency loss can be introduced to control the high-frequency components in the generated perturbations. High-frequency perturbations typically contain details and local variations, and are prone to failure across different inputs and models. Therefore, penalizing the high-frequency components helps generate more general perturbations and improves the transferability of the attack.

[0111] Specifically, the method 100 further includes: after generating adversarial image samples, obtaining the high-frequency components of the image samples and constraining the high-frequency components.

[0112] High-frequency loss refers to controlling the frequency components of the perturbation so that the perturbation is mainly concentrated in the low-frequency region. Low-frequency perturbations usually correspond to large-scale changes in the image and have strong transferability, while high-frequency components affect local details and are easily affected by the diversity of models and data. In the frequency domain, the low-frequency components of the perturbation are relatively smooth and widely applicable, while the high-frequency components are usually concentrated on details and noise.

[0113] To achieve this goal, after obtaining the adversarial examples, the generated perturbation images (perturbation samples) are first transformed to the frequency domain using either a Fourier Transform or a Discrete Cosine Transform (DCT). The low-frequency components in the frequency domain contain the macroscopic changes of the perturbation, while the high-frequency components correspond to the details and rapid changes in the image. By penalizing and constraining the high-frequency components, the generation of detail perturbations can be reduced, thus making the perturbation behave more consistently across different images and models.

[0114] The formula for realizing high-frequency loss is expressed as follows:

[0115]

[0116] in, This represents the generated perturbation image. It is a perturbation image Fourier transform, This is a masking function used to extract low-frequency regions, where i and j are the row and column numbers of the image pixels, respectively. Here, the mask... The low-frequency region is set to 0 (preserving low frequencies), while the high-frequency component is assigned a non-zero value, allowing for specific penalty for high-frequency components. Specifically, It is a binary mask where the value is 0 in the low-frequency region and 1 in the high-frequency region. This mask function can effectively isolate low-frequency and high-frequency components and impose a penalty on the high-frequency components.

[0117] like Figure 5 As shown, the present invention provides a multi-target adversarial attack device 500 based on text description, comprising: a first extraction module 501, used to extract text features from target text based on a text encoder in a preset model, wherein the target text has a target category and the target category does not correspond to the target text; a second extraction module 502, used to extract image features from the original image based on an image encoder in a preset model; a fusion module 503, used to fuse text features and image features based on a semantic fusion module in a preset model to obtain fused semantic features; a generation module 504, used to generate adversarial perturbation data images from the semantic features based on a generator in a preset model; and an adding module 505, used to add the adversarial perturbation data images to the original image to generate corresponding adversarial image samples.

[0118] Specifically, the fusion module 503 includes: a transformation unit, used to embed text features as queries and use the latent encoding of text features as keys and values ​​using linear transformations in the semantic fusion model to determine attention maps in spatial and channel dimensions; and a mapping unit, used to determine the parameters of the linear transformation based on the attention maps and use the parameters to map text features to the latent subspace of the corresponding preset model, wherein the latent subspace is a subspace of image features, so as to fuse text features with image features.

[0119] Specifically, the generation module 504 is used to: generate an adversarial perturbation image based on the text feature distribution of the target category, so as to shift the semantic features of the adversarial image sample to the feature center of the target category through the adversarial perturbation image.

[0120] Specifically, the generation module 504 is used to: decode semantic features based on the decoder in the generator to obtain an adversarial image.

[0121] In addition, the device 500 also includes: a determination module, specifically used to generate a composite loss function based on the cross-entropy loss function, the KL divergence loss function, and a regularization term; and a generator based on the composite loss function.

[0122] The pre-built models include the contrastive language-image pre-training CLIP framework.

[0123] In addition, the device 500 also includes: a third extraction module for extracting image features from the original image using the encoder in the generator; a stitching module for stitching the semantic features with the extracted image features after obtaining the fused semantic features; and a generation module 504 for decoding the stitched image features using the decoder in the generator to obtain an image resistant to perturbation.

[0124] In addition, the device 500 further includes: a receiving module, used to receive an image of a first preset size for carrying text features and an image of a first preset size for carrying image features through a multi-scale fusion module in a preset model, and to generate a text feature image and three feature images of different scales corresponding to the image feature image through adaptive pooling operations; a copying module, used to copy the three feature images of the corresponding different scales along the spatial dimension to generate a feature image of the first preset size for the text feature image and the image feature image respectively; a receiving module, used to receive the original image and use a lightweight network to extract features from the original image to determine the fusion weights of the multi-scale features and apply them to feature maps of different scales; and a weighted fusion module, used to perform weighted fusion on the three feature images of the corresponding different scales and the corresponding feature images of the first preset size based on the fusion weights to obtain a feature image of the second preset size, so as to fuse text features and image features.

[0125] In addition, the device 500 also includes a constraint module for obtaining the high-frequency components of the image samples after generating adversarial image samples and constraining the high-frequency components.

[0126] Other embodiments of the device 500 can be found in the method embodiments described above, which correspond to the method 100 embodiment, and will not be repeated here. Furthermore, it should be understood that the device modules involved in this device 500 may differ from those involved in method 100.

[0127] like Figure 6 As shown in the illustration, this application also provides a multi-target adversarial attack device 600 based on text description, including: a memory 601 and a processor 602. The processor 602 reads a computer program from the memory 601 and performs the following operations:

[0128] Text features are extracted from the target text using a text encoder in a pre-built model. The target text has a target category, but the target category does not correspond to the target text. Image features are extracted from the original image using an image encoder in a pre-built model. The text features and image features are fused using a semantic fusion module in a pre-built model to obtain fused semantic features. An adversarial perturbation image is generated using a generator in a pre-built model based on the semantic features. The adversarial perturbation image is added to the original image to generate corresponding adversarial image samples.

[0129] Other embodiments of the device 600 can be found in the embodiments of the apparatus 500 and method 100 described above, which correspond to the embodiments of the apparatus 500 and method 100, and will not be repeated here.

[0130] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process may be rearranged without departing from the scope of this disclosure. The appended method claims provide elements of various steps in an exemplary order and are not intended to limit the scope to the specific order or hierarchy described.

[0131] To make the description of this disclosure more detailed and complete, illustrative descriptions of the embodiments and specific examples of the present invention have been provided above; however, this is not the only form of implementing or utilizing the specific examples of the present invention. The embodiments cover the features of multiple specific examples and the method steps and their order for constructing and operating these specific examples. However, other specific examples may also be used to achieve the same or equivalent functions and order of steps.

[0132] In the detailed description above, various features are combined together in a single embodiment to simplify this disclosure. This approach to disclosure should not be construed as reflecting an intention that embodiments of the claimed subject matter require more features than are explicitly stated in each claim. Rather, as reflected in the appended claims, the invention is presented with fewer features than all of the features in a single disclosed embodiment. Therefore, the appended claims are hereby explicitly incorporated into the detailed description, with each claim representing a separate preferred embodiment of the invention.

[0133] The disclosed embodiments have been described above to enable any person skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments without departing from the spirit and scope of this disclosure. Therefore, this disclosure is not limited to the embodiments given herein, but is consistent with the broadest scope of the principles and novel features disclosed in this application.

[0134] The foregoing description includes examples of one or more embodiments. It is certainly impossible to describe all possible combinations of components or methods in order to describe the above embodiments, but those skilled in the art will recognize that further combinations and arrangements of the various embodiments are possible. Therefore, the embodiments described herein are intended to cover all such changes, modifications, and variations that fall within the scope of the appended claims. Furthermore, the term "comprising" as used in the specification or claims is interpreted in a manner similar to the term "including," as interpreted when used as a conjunction in the claims. Additionally, the use of any term "or" in the specification of the claims is intended to mean "non-exclusive or."

[0135] Those skilled in the art will also understand that the various illustrative logical blocks, units, and steps listed in the embodiments of the present invention can be implemented by electronic hardware, computer software, or a combination of both. To clearly demonstrate the interchangeability of hardware and software, the functions of the various illustrative components, units, and steps described above have been generally described. Whether such functionality is implemented through hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the described functions using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of the present invention.

[0136] The various illustrative logic blocks or units described in the embodiments of this invention can be implemented or operate the described functions using a general-purpose processor, digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The general-purpose processor can be a microprocessor; alternatively, it can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented using a combination of computing devices, such as a digital signal processor and a microprocessor, multiple microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.

[0137] The steps of the methods or algorithms described in the embodiments of this invention can be directly embedded in hardware, a software module executed by a processor, or a combination of both. The software module can be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from and write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and storage medium can be housed in an ASIC, which can be housed in a user terminal. Optionally, the processor and storage medium can also be housed in different components of the user terminal.

[0138] In one or more exemplary designs, the functions described in the embodiments of the present invention can be implemented in hardware, software, firmware, or any combination of these three. If implemented in software, these functions can be stored on a computer-readable medium or transmitted on a computer-readable medium in the form of one or more instructions or code. Computer-readable media include computer storage media and communication media that facilitate the transfer of computer programs from one place to another. Storage media can be any available media that can be accessed by a general-purpose or special-purpose computer. For example, such computer-readable media can include, but is not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store program code in the form of instructions or data structures and other forms that can be read by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Furthermore, any connection can be suitably defined as a computer-readable medium, for example, if the software is transmitted from a website, server, or other remote resource via a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wirelessly, such as infrared, wireless, and microwave, it is also included in the defined computer-readable medium. The disks and discs mentioned include compressed disks, laser discs, optical discs, DVDs, floppy disks, and Blu-ray discs. Disks typically copy data magnetically, while discs typically copy data optically using lasers. Combinations of the above can also be contained in computer-readable media.

[0139] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-target adversarial attack method based on text description, characterized in that, include: Text features are extracted from the target text based on the text encoder in the pre-set model, and the target text has a corresponding target category; Image features are extracted from the original image based on the image encoder in the preset model, wherein the category of the original image is inconsistent with the target category; Based on the semantic fusion module in the pre-built model, the text features and the image features are fused to obtain the fused semantic features; Based on the generator in the pre-set model, the semantic features are used to generate an adversarial perturbation image; Add the adversarial perturbation image to the original image to generate corresponding adversarial image samples; The semantic fusion module is set in the generator; The method further includes: A composite loss function is generated based on the cross-entropy loss function, the KL divergence loss function, and the regularization term. The generator is generated based on the composite loss function; The step of generating an adversarial perturbation feature image from the semantic features based on the generator in the preset model includes: Based on the text feature distribution of the target category, the generator generates an adversarial perturbation image from the semantic features, so as to shift the semantic features of the adversarial image sample to the feature center of the target category through the adversarial perturbation image.

2. The method according to claim 1, characterized in that, The semantic fusion module based on the pre-set model fuses the text features and the image features to obtain fused semantic features, including: By utilizing linear transformations in a semantic fusion model, textual features are embedded as queries, and the latent encodings of these textual features are used as keys and values ​​to determine attention maps in spatial and channel dimensions. Based on the attention map, the parameters of the linear transformation are determined, and the text features are mapped to the latent subspace of the corresponding preset model using the parameters. The latent subspace is a subspace of image features, so that the text features and image features are fused.

3. The method according to claim 1, characterized in that, The step of generating an adversarial perturbation image based on the semantic features using the generator in the pre-set model includes: Based on the decoder in the generator, the semantic features are decoded to obtain an image that resists perturbation.

4. The method according to claim 1, characterized in that, The pre-built model includes the contrastive language-image pre-training CLIP framework.

5. The method according to claim 1, characterized in that, The method further includes: The encoder in the generator is used to extract image features from the original image; After obtaining the fused semantic features, the semantic features are concatenated with the extracted image features; The image against perturbation is obtained by decoding the features of the stitched image using the decoder in the generator.

6. The method according to claim 1, characterized in that, The method further includes: The multi-scale fusion module in the preset model receives an image of a first preset size for carrying the text features and an image of a first preset size for carrying the image features, and generates a text feature image and three feature images of different scales corresponding to the image feature image through an adaptive pooling operation. For text feature images and image feature images respectively, the three corresponding feature images at different scales are copied along the spatial dimension to generate a feature image of the first preset size; The original image is received, and a lightweight network is used to extract features from the original image to determine the fusion weights of multi-scale features, which are then applied to feature maps at different scales. For text feature images and image feature images respectively, the corresponding three feature images of different scales and the corresponding feature images of the first preset size are weighted and fused based on the fusion weight to obtain the feature image of the second preset size, so as to fuse text features and image features.

7. The method according to claim 1, characterized in that, The method further includes: After generating the adversarial image sample, the high-frequency components of the image sample are obtained and constrained.

8. A multi-target adversarial attack device based on text description, characterized in that, include: The first extraction module is used to extract text features from the target text based on the text encoder in the preset model, wherein the target text has a corresponding target category; The second extraction module is used to extract image features from the original image based on the image encoder in the preset model, wherein the category of the original image is inconsistent with the target category; The fusion module is used to fuse the text features and the image features based on the semantic fusion module in the preset model to obtain the fused semantic features; The generation module is used to generate an adversarial perturbation image based on the generator in the preset model from the semantic features; An adding module is used to add the adversarial perturbation image to the original image to generate corresponding adversarial image samples; The semantic fusion module is set in the generator; The device further includes: a determining module, used for... A composite loss function is generated based on the cross-entropy loss function, the KL divergence loss function, and the regularization term. The generator is generated based on the composite loss function; The generation module is specifically used to: generate an adversarial perturbation image based on the text feature distribution of the target category, so as to shift the semantic features of the adversarial image sample to the feature center of the target category through the adversarial perturbation image.

Images