Multi-modal adversarial sample generation system and method based on general disturbance characterization learning
Through a multimodal adversarial sample generation method based on universal perturbation representation learning, the problems of low efficiency and insufficient transferability of multimodal adversarial sample generation are solved, and the versatility and robustness of adversarial attacks are improved, which is suitable for security assessment and defense of multimodal models.
Patent Information
- Application Number
- CN202510672312.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-23
- Publication Date
- 2025-09-12
AI Technical Summary
Existing multimodal adversarial sample generation technology lacks portability in cross-modal interaction scenarios, resulting in a sharp drop in attack success rate, high computational overhead, and poor adaptability.
A method based on universal perturbation representation learning is adopted to initialize perturbations through multimodal datasets and alternative models. Intra-modal and inter-modal contrastive learning is combined to optimize adversarial perturbations and generate adversarial samples that can produce deep perturbations in the multimodal embedding space.
It improves the versatility and robustness of adversarial attacks, can significantly destroy multimodal semantic embedding relationships, reduce attack complexity, and improve the security and robustness of multimodal models in open environments.
Smart Images
Figure CN120635627A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of intelligent security technology, and in particular to a multimodal adversarial sample generation system and method based on universal perturbation representation learning. Background Art
[0002] Simply put, adversarial examples involve adding very small perturbations to the input image, causing it to influence the model's predictions. While these perturbations can be imperceptible to the human visual system, they can completely alter the model's output. With the widespread adoption of multimodal intelligent systems powered by deep learning in critical scenarios such as autonomous driving and financial security, the security threats they face are becoming increasingly complex. Adversarial example attacks targeting multimodal foundational models have become a research focus. These attacks, by constructing adversarial examples, mislead the model into making incorrect decisions, exposing serious vulnerabilities.
[0003] Existing adversarial attack methods are mostly focused on unimodal scenarios, making them difficult to effectively transfer to cross-modal interaction scenarios. Furthermore, most methods require generating perturbations for each input individually, resulting in high computational overhead and poor adaptability. Existing adversarial sample generation techniques for multimodal scenarios suffer from insufficient transferability during black-box attacks, leading to a sharp drop in attack success rates. To address this issue, a universal adversarial perturbation generation method that is input-independent, transferable, and capable of significantly disrupting multimodal semantic embeddings is urgently needed to evaluate and improve the security and robustness of current multimodal models in open environments. Summary of the Invention
[0004] In view of the above problems, the present invention provides a multimodal adversarial sample generation system and method based on universal perturbation representation learning, which solves the technical problems of low efficiency and poor performance caused by insufficient transferability of multimodal adversarial sample generation in the prior art.
[0005] In one aspect, the present invention provides a method for generating multimodal adversarial samples based on universal perturbation representation learning, comprising the following steps:
[0006] Step S1: determining a multimodal dataset and a replacement model, wherein the multimodal dataset includes a plurality of samples consisting of images and texts;
[0007] Step S2: performing a disturbance initialization process based on the image in the sample in the multimodal dataset and the replacement model to obtain an initial value of the anti-disturbance;
[0008] Step S3: Collect a sample from the multimodal dataset, construct the sample based on the adversarial perturbation, perform feature extraction based on the substitution model, and obtain feature vectors of the anchor sample, positive sample, and negative sample of the image in the sample and feature vector of the negative sample of the text in the sample;
[0009] Step S4: performing intra-modal and inter-modal comparative learning based on the feature vectors of the anchor samples, positive samples, and negative samples of the image and the feature vectors of the negative samples of the text to obtain a multimodal loss, and updating the adversarial perturbation based on the multimodal loss;
[0010] Return to step S3 until the multimodal loss converges, and finally output the adversarial perturbation;
[0011] Step S5: superimpose the adversarial perturbation on any image in the sample to obtain an adversarial sample.
[0012] Preferably, step S2 specifically includes:
[0013] Step S2-1, randomly generate universal adversarial perturbations;
[0014] Step S2-2: Collect a sample from the multimodal dataset, and superimpose the universal adversarial perturbation on the image in the sample to obtain an adversarial image;
[0015] Step S2-3: input the image in the sample and the adversarial image into the substitution model respectively, obtain feature vectors of the image and the adversarial image, calculate the initialization loss based on the feature vectors of the image and the adversarial image, and update the universal adversarial perturbation;
[0016] Step S2-4: Return to step S2-2 until the initialization loss converges, and determine the updated universal adversarial perturbation as the initial value of the adversarial perturbation.
[0017] Preferably, the step S2-3 of calculating the initialization loss based on the feature vectors of the image and the adversarial image and updating the universal adversarial perturbation specifically includes:
[0018] The universal adversarial perturbation is updated using the fast gradient sign method
[0019]
[0020] Among them, δ v,ori represents the initial universal adversarial perturbation, x v represents a clean image, Represents the clean image superposition δ v,ori The adversarial sample obtained, ∈ v represents the size limit of the adversarial perturbation on the image, J(·) represents the initialization loss function, and F i (·) represents the feature vector representation of the image encoder output of the alternative model, Represents an alternative model about The gradient of , sign(·) represents the sign function.
[0021] Preferably, step S3 specifically includes:
[0022] Step S3-1: for one sample, superimpose the adversarial perturbation on the image in the sample to obtain an anchor sample of the image;
[0023] Step S3-2: randomly crop the image in the sample and scale it to the original image size, and superimpose the adversarial perturbation on the obtained image to obtain a positive sample of the image;
[0024] Step S3-3: Randomly select a data enhancement algorithm from multiple data enhancement algorithms and apply it to images in the sample, repeat multiple times to obtain multiple negative samples;
[0025] Step S3-4: input the anchor sample, positive sample, and negative sample of the image into the substitution model to obtain feature vectors of the anchor sample, positive sample, and negative sample of the image;
[0026] Step S3-5: Input the text in the sample into the substitution model to obtain a feature vector of the text as a feature vector of a negative sample of the text.
[0027] Preferably, step S4 specifically includes:
[0028] Step S4-1, calculating the intra-modality loss based on the feature vectors of the anchor sample, positive sample, and negative sample of the image;
[0029] Step S4-2, calculating the inter-modality loss using the anchor sample and positive sample of the image and the negative sample of the text;
[0030] Step S4-3, obtaining the multimodal loss from the intra-modal loss and the inter-modal loss;
[0031] Step S4-4: Update the adversarial perturbation using the multimodal loss.
[0032] Preferably, the calculation expression of the intra-modal loss is:
[0033]
[0034] in, represents the intra-modal loss, is the InfoNCE loss function, represents the adversarial sample as the anchor sample, represents the adversarial sample obtained after random cropping as a positive sample, x v +x ′ v Construct a clean image as a negative sample and its randomly data-augmented image;
[0035]
[0036] Among them, q is the value of the anchor sample corresponding to the input, k + is the value of the positive sample corresponding to the input, k - is the value of the negative sample, τ is the temperature coefficient, exp(·) represents the calculation of the natural exponential, i is the negative sample index, Indicates the value of the input corresponding to the i-th negative sample;
[0037] The calculation expression of the inter-modal loss is:
[0038]
[0039] Among them, F t (·) represents the feature vector representation of the text output of the text encoder of the alternative model, x t Represents clean text as negative sample;
[0040] The calculation expression of the multimodal loss is:
[0041]
[0042] in, represents the multimodal loss, δ v represents adversarial perturbation, ∈ v represents the size limit of the adversarial perturbation, and α is the control inter-modal loss The hyperparameter of weight, st‖δ v ‖ ∞ ≤∈ v Indicates that the anti-perturbation δ v The absolute value of all dimensions of does not exceed ∈ v ;
[0043] The calculation expression for updating the adversarial disturbance by the multimodal loss is:
[0044]
[0045] in, Represents a clean image superimposed with universal adversarial perturbation δ v The adversarial examples obtained are Represents an alternative model about gradient.
[0046] Preferably, the multimodal dataset is the Flickr30k test set; the alternative model is the Contrastive Language-Image Pre-training model; the initialization loss function J(·) is the KL divergence loss function; the perturbation size limit ∈ v Set to 12 / 255; controls intermodal loss The hyperparameter α of the weight is set to 3.
[0047] In one aspect, the present invention provides a multimodal adversarial sample generation system based on universal perturbation representation learning, wherein a determination module 10 is configured to determine a multimodal dataset and a replacement model, wherein the multimodal dataset includes a plurality of samples consisting of images and texts;
[0048] An initialization module 20 is configured to perform a perturbation initialization process based on images in samples in the multimodal dataset and the replacement model to obtain an initial value for counteracting the perturbation;
[0049] An extraction module 30 is configured to collect a sample from a multimodal dataset, construct the sample based on the adversarial perturbation, perform feature extraction based on the substitution model, and obtain feature vectors of anchor samples, positive samples, and negative samples of the image in the sample and feature vectors of negative samples of the text in the sample;
[0050] a multimodal learning module 40 for performing intra-modal and inter-modal comparative learning based on feature vectors of anchor samples, positive samples, and negative samples of images and feature vectors of negative samples of text, obtaining a multimodal loss, updating the adversarial perturbation based on the multimodal loss, until the multimodal loss converges, and finally outputting the adversarial perturbation;
[0051] The superposition module 50 is configured to superimpose the adversarial perturbation on any image in the sample to obtain an adversarial sample.
[0052] Compared with the prior art, the present invention has at least the following beneficial effects:
[0053] (1) The present invention effectively improves the versatility and robustness of adversarial attacks by providing a two-stage perturbation optimization mechanism. In the first stage, an initial universal adversarial perturbation is obtained by optimizing the training set using an alternative model. This perturbation can produce a stable interference effect on multiple images and has good universality. In the second stage, based on this initial perturbation, combined with contrastive learning ideas and methods, positive and negative samples are constructed using image enhancement, cropping, etc., and feature contrast losses within and between modalities are integrated to guide the perturbation to produce deeper interference in the multimodal embedding space. This two-stage method of first optimizing the initial perturbation and then contrast-optimizing the final perturbation improves the stability of the perturbation and its cross-sample adaptability.
[0054] (2) The present invention inputs anchor samples (original images + perturbations), cropped images (positive samples), and enhanced image sets (negative samples) into an image encoder, calculates intra-modal contrast loss, and compares the image embeddings (perturbed anchor samples and positive samples) with the original text embeddings across modalities, further disrupting the model's image-text semantic alignment process, thereby achieving the purpose of interfering with multimodal semantic understanding. This allows the adversarial perturbation to not only affect the image representation of the visual model, but also interfere with the alignment relationship between language and vision, thereby improving the performance of the adversarial perturbation cross-model transfer attack.
[0055] (3) Since multimodal models such as CLIP are widely used in task scenarios such as image-text matching, image generation, and text-guided retrieval, the present invention generates a single perturbation that can act on all inputs, significantly reducing the complexity of the attack. This invention provides a new paradigm for security assessment and active defense of multimodal basic models. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] The drawings are only for purposes of illustrating particular embodiments and are not to be considered limiting of the invention.
[0057] Figure 1 Flowchart of the multimodal adversarial sample generation method for universal perturbation representation learning provided by the present invention.
[0058] Figure 2 Detailed flowchart of the multimodal adversarial sample generation method for universal perturbation representation learning provided by the present invention.
[0059] Figure 3 Schematic diagram of the structure of the multimodal adversarial sample generation system for universal perturbation representation learning provided by the present invention.
[0060] Figure 4 This is an example diagram of the adversarial sample finally generated by the present invention.
[0061] Figure numerals: 10 - determination module, 20 - initialization module, 30 - extraction module, 40 - multimodal learning module, 50 - superposition module. DETAILED DESCRIPTION
[0062] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. In addition, the present invention can also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited by the specific embodiments disclosed below.
[0063] In order to illustrate the effectiveness of the method proposed by the present invention, the above technical solution of the present invention is described in detail below through a specific embodiment. Figure 1 、 Figure 2 As shown in the figure, a multimodal adversarial sample generation method based on universal perturbation representation learning is disclosed. The specific implementation steps are as follows:
[0064] Step S1: determining a multimodal dataset and a replacement model, wherein the multimodal dataset includes a plurality of samples consisting of images and texts;
[0065] The multimodal adversarial sample generation task of the present invention is targeted at different datasets. In this step, it is necessary to determine the multimodal dataset targeted by the adversarial sample generation task. The multimodal dataset contains multiple samples, each of which includes data from multiple different modalities.
[0066] In some embodiments, each sample in the multimodal dataset may include data combined from multiple modalities such as image, voice, video, or text, such as an image-text pair, an image-voice-text pair, and the like.
[0067] Since the target model of the adversarial task may be an inaccessible or unmodifiable neural network model, such as a commercial API, the present invention adopts a substitute model. The substitute model of the present invention refers to an accessible model whose structure and behavior are as close as possible to the target model. It is used to generate adversarial samples in the adversarial task. In this way, the adversarial task can be completed and applicability can be guaranteed under the premise that the target model is difficult to access.
[0068] The substitution model of the present invention is a multimodal model, which can accept a variety of different modal data as input, perform feature embedding processing (or encoding) on the different modal data, and obtain feature vectors that represent the characteristics of the different modal data.
[0069] In some embodiments, the multimodal model may be a multimodal model including an image encoder and a text encoder, which performs image and text encoding on the input image-text pair respectively to obtain feature vectors of the image and text.
[0070] By adopting a multimodal model as an alternative model in the above manner, the present invention can be applied to cross-modal interaction scenarios, avoiding the limitation of a single modality in a single-modal scenario adversarial task.
[0071] Step S2: Perform perturbation initialization processing based on the images in the samples in the multimodal dataset and the replacement model to obtain the initial value of the anti-perturbation; the specific steps are as follows.
[0072] Step S2-1, randomly generate universal adversarial perturbations;
[0073] In this step, the present invention may use random noise as the initial value of the universal anti-disturbance.
[0074] In some embodiments, a noise tensor with the same size as the image can be generated as the initial value of the universal adversarial perturbation by random sampling based on a preset amplitude range (such as [-ε, ε], where ε is the upper limit of the perturbation) based on a random distribution such as a uniform distribution or a Gaussian distribution.
[0075] Step S2-2: for one of the samples, superimpose the universal adversarial perturbation on the image in the sample to obtain an adversarial image;
[0076] In this step, a sample is randomly selected from the multimodal dataset, and an image in the sample is selected. The universal adversarial perturbation is superimposed on the image to obtain an adversarial image for subsequent perturbation optimization.
[0077] Step S2-3: input the image in the sample and the adversarial image into the substitution model respectively, obtain feature vectors of the image and the adversarial image, calculate the initialization loss based on the feature vectors of the image and the adversarial image, and update the universal adversarial perturbation;
[0078] In this step, the clean image and the adversarial image in the sample are input into the substitution model respectively to obtain the feature vectors of the clean image and the adversarial image, and then the fast gradient sign method is used to update the universal adversarial perturbation. The expression is:
[0079]
[0080] Among them, δ v,ori represents the initial universal adversarial perturbation, x v represents a clean image, Represents the clean image superposition δ v,ori The adversarial sample obtained, ∈ v represents the size limit of the adversarial perturbation on the image, J(·) represents the initialization loss function, and F i (·) represents the feature vector representation of the image encoder output of the alternative model, Represents an alternative model about The gradient of , sign(·) represents the sign function.
[0081] In some embodiments, the initialization loss function J(·) of the present invention can adopt the KL divergence loss function to calculate the loss between the clean image and the adversarial image feature vector representation.
[0082] Step S2-4: Return to step S2-2 until the initialization loss converges, and determine the updated universal adversarial perturbation as the initial value of the adversarial perturbation.
[0083] Through the above steps, the present invention iteratively updates the randomly generated universal adversarial perturbation based on the difference in feature vectors of multiple sampled clean images and adversarial images in the multimodal dataset, and finally generates an image-independent universal adversarial perturbation for the multimodal dataset as the initial value for subsequent optimization.
[0084] In some embodiments, the process ends when the initialization loss is less than a preset value or when the maximum number of iterations is reached. The maximum number of iterations in this step may be 100.
[0085] Step S3: Collect a sample from the multimodal dataset, construct the sample based on the adversarial perturbation, perform feature extraction based on the substitution model, obtain the feature vectors of the anchor sample, positive sample and negative sample of the image in the sample, and the feature vector of the negative sample of the text in the sample, as described in detail below.
[0086] Step S3-1: for one sample, superimpose the adversarial perturbation on the image in the sample to obtain an anchor sample of the image;
[0087] Step S3-2: randomly crop the image in the sample and scale it to the original image size, and superimpose the adversarial perturbation on the obtained image to obtain a positive sample of the image;
[0088] Step S3-3: Randomly select a data enhancement algorithm from multiple data enhancement algorithms and apply it to images in the sample, repeat multiple times to obtain multiple negative samples;
[0089] In some embodiments, the present invention selects 11 data enhancement algorithms, and randomly selects one of them to generate a negative sample each time a negative sample is obtained.
[0090] Step S3-4: input the anchor sample, positive sample, and negative sample of the image into the substitution model to obtain feature vectors of the anchor sample, positive sample, and negative sample of the image;
[0091] Step S3-5: input the text in the sample into the substitution model to obtain a feature vector of the text as a feature vector of a negative sample of the text;
[0092] Through the above steps, the present invention constructs a series of multimodal anchor samples, positive samples and negative samples for multimodal contrastive learning, which are used for further optimization of anti-perturbation in subsequent steps.
[0093] Step S4: performing intra-modal and inter-modal comparative learning based on the feature vectors of the anchor samples, positive samples, and negative samples of the image and the feature vectors of the negative samples of the text to obtain a multimodal loss, and updating the adversarial perturbation based on the multimodal loss;
[0094] Step S4-1, calculating the intra-modality loss based on the feature vectors of the anchor sample, positive sample, and negative sample of the image;
[0095] The calculation expression for this step is:
[0096]
[0097] in, represents the intra-modal loss, is the InfoNCE loss function, represents the adversarial sample as the anchor sample, represents the adversarial sample obtained after random cropping as a positive sample, x v +x′ v Construct clean images as negative samples and their randomly augmented images.
[0098] The expression of InfoNCE loss function is:
[0099]
[0100] Among them, q is the value of the anchor sample corresponding to the input, k + is the value of the positive sample corresponding to the input, k - is the value of the negative sample, τ is the temperature coefficient, exp(·) represents the calculation of the natural exponential, i is the negative sample index, Indicates the value passed in corresponding to the i-th negative sample.
[0101] Step S4-2, calculating the inter-modality loss using the anchor sample and positive sample of the image and the negative sample of the text;
[0102] The calculation expression for this step is:
[0103]
[0104] Among them, F t (·) represents the feature vector representation of the text output of the text encoder of the alternative model, x t Represents the clean text as negative sample.
[0105] Step S4-3: Obtain the multimodal loss from the intra-modal loss and the inter-modal loss.
[0106] The calculation expression for this step is:
[0107]
[0108] in, represents the multimodal loss, δ v represents adversarial perturbation, ∈ vrepresents the size limit of the adversarial perturbation, and α is the control inter-modal loss The hyperparameter of weight, st‖δ v ‖ ∞ ≤∈ v Indicates that the universal adversarial perturbation δ v The absolute value of all dimensions of does not exceed ∈ v .
[0109] Step S4-4: Update the adversarial perturbation using the multimodal loss.
[0110] The calculation expression for this step is:
[0111]
[0112] in, Represents a clean image superimposed with universal adversarial perturbation δ v The adversarial examples obtained are Represents an alternative model about gradient.
[0113] Through the above steps, the present invention constructs a contrast loss function based on the features between images (intra-modality) and between images and text (inter-modality), so that the perturbation optimization process considers not only the feature comparison of the image itself but also the feature comparison between modalities, so that the adversarial perturbation can not only affect the image representation of the visual model, but also interfere with the alignment relationship between language and vision.
[0114] Then return to step S3 until the multimodal loss converges, and finally output the adversarial perturbation.
[0115] In this step, after multiple iterations, the final adversarial perturbation is obtained when the multimodal loss is less than a preset value or the maximum number of iterations is reached. The maximum number of iterations in this step can be 100.
[0116] Step S5: superimpose the adversarial perturbation on any image in the sample to obtain an adversarial sample.
[0117] The adversarial sample obtained by superimposing adversarial disturbances in the present invention is as follows Figure 4 shown.
[0118] The present invention also provides a multimodal adversarial sample generation system based on universal perturbation representation learning, such as Figure 3 As shown, it includes the following modules:
[0119] A determination module 10 is configured to determine a multimodal dataset and a replacement model, wherein the multimodal dataset includes a plurality of samples consisting of images and texts;
[0120] An initialization module 20 is configured to perform a perturbation initialization process based on images in samples in the multimodal dataset and the replacement model to obtain an initial value for counteracting the perturbation;
[0121] An extraction module 30 is configured to collect a sample from a multimodal dataset, construct the sample based on the adversarial perturbation, perform feature extraction based on the substitution model, and obtain feature vectors of anchor samples, positive samples, and negative samples of the image in the sample and feature vectors of negative samples of the text in the sample;
[0122] The multimodal learning module 40 is used to perform intra-modal and inter-modal comparative learning based on the feature vectors of the anchor samples, positive samples and negative samples of the image and the feature vectors of the negative samples of the text, obtain the multimodal loss, and update the adversarial perturbation based on the multimodal loss; until the multimodal loss converges, and finally output the adversarial perturbation.
[0123] The superposition module 50 is configured to superimpose the adversarial perturbation on any image in the sample to obtain an adversarial sample.
[0124] The present invention also provides a specific embodiment of a multimodal adversarial sample generation method based on universal perturbation representation learning, which is described in detail as follows.
[0125] Example 1
[0126] In order to verify the effectiveness and practicality of the present invention, the image-text retrieval task is selected as a multimodal scenario. According to step S1, the Flickr30k test set is selected as the dataset (5000 image-text pairs) according to the task, and CLIP is used as the alternative model. According to steps S2-S4, the number of negative samples constructed by the random data augmentation algorithm in intra-modal contrastive learning is 8, and the number of negative samples constructed by text in inter-modal contrastive learning is 5. A total of 100 iterations are trained, and finally the generated universal adversarial perturbation is saved and superimposed on all images in the dataset to form an adversarial sample dataset.
[0127] Preferably, the maximum number of iterations of the iterative process in steps S2-4 and S4 is set to 100.
[0128] Preferably, the disturbance size limit ∈ in step S2-3 and step S4-3 is v Set to 12 / 255.
[0129] Preferably, the hyperparameter α for controlling the inter-modality contrastive learning loss weight in step S4-3 is set to 3.
[0130] Using the generated adversarial sample dataset, a white-box attack was performed on the CLIP model, and a black-box attack was performed on the ALBEF model. The attack success rate of the text retrieval task based on image under the white-box attack was 99.63%, and the attack success rate of the image retrieval task based on text was 99.54%. The attack success rate of the text retrieval task based on image under the black-box attack was 65.07%, and the attack success rate of the image retrieval task based on text was 73.88%, which are relatively good results, indicating that the present invention is effective and feasible.
[0131] Although the specific embodiments of the present invention have been described in a particular order, it should be understood that such actions or steps are required to be performed in the particular order shown or in a sequential order, or that all illustrated actions or steps are required to be performed to obtain the desired result. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be construed as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination.
[0132] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.
Claims
1. A multimodal adversarial sample generation method based on universal perturbation representation learning, characterized by: The following steps are involved: Step S1: determining a multimodal dataset and a replacement model, wherein the multimodal dataset includes a plurality of samples consisting of images and texts; Step S2: performing a disturbance initialization process based on the image in the sample in the multimodal dataset and the replacement model to obtain an initial value of the anti-disturbance; Step S3: Collect a sample from the multimodal dataset, construct the sample based on the adversarial perturbation, perform feature extraction based on the substitution model, and obtain feature vectors of the anchor sample, positive sample, and negative sample of the image in the sample and feature vector of the negative sample of the text in the sample; Step S4: performing intra-modal and inter-modal comparative learning based on the feature vectors of the anchor samples, positive samples, and negative samples of the image and the feature vectors of the negative samples of the text to obtain a multimodal loss, and updating the adversarial perturbation based on the multimodal loss; Return to step S3 until the multimodal loss converges, and finally output the adversarial perturbation; Step S5: superimpose the adversarial perturbation on any image in the sample to obtain an adversarial sample.
2. The multimodal adversarial sample generation method based on universal perturbation representation learning according to claim 1 is characterized in that: Step S2 specifically includes: Step S2-1, randomly generate universal adversarial perturbations; Step S2-2: Collect a sample from the multimodal dataset, and superimpose the universal adversarial perturbation on the image in the sample to obtain an adversarial image; Step S2-3: input the image in the sample and the adversarial image into the substitution model respectively, obtain feature vectors of the image and the adversarial image, calculate the initialization loss based on the feature vectors of the image and the adversarial image, and update the universal adversarial perturbation; Step S2-4: Return to step S2-2 until the initialization loss converges, and determine the updated universal adversarial perturbation as the initial value of the adversarial perturbation.
3. The multimodal adversarial sample generation method based on universal perturbation representation learning according to claim 2 is characterized in that: Calculating the initialization loss based on the feature vectors of the image and the adversarial image and updating the universal adversarial perturbation in step S2-3 specifically includes: The universal adversarial perturbation is updated using the fast gradient sign method Among them, δ v,ori represents the initial universal adversarial perturbation, x v represents a clean image, Represents the clean image superposition δ v,ori The adversarial sample obtained, ∈ v represents the size limit of the adversarial perturbation on the image, J(·) represents the initialization loss function, and F i (·) represents the feature vector representation of the image encoder output of the alternative model, Represents an alternative model about The gradient of , sign(·) represents the sign function.
4. The multimodal adversarial sample generation method based on universal perturbation representation learning according to claim 3 is characterized in that Step S3 specifically includes: Step S3-1: for one sample, superimpose the adversarial perturbation on the image in the sample to obtain an anchor sample of the image; Step S3-2: randomly crop the image in the sample and scale it to the original image size, and superimpose the adversarial perturbation on the obtained image to obtain a positive sample of the image; Step S3-3: Randomly select a data enhancement algorithm from multiple data enhancement algorithms and apply it to images in the sample, repeat multiple times to obtain multiple negative samples; Step S3-4: input the anchor sample, positive sample, and negative sample of the image into the substitution model to obtain feature vectors of the anchor sample, positive sample, and negative sample of the image; Step S3-5: Input the text in the sample into the substitution model to obtain a feature vector of the text as a feature vector of a negative sample of the text.
5. The multimodal adversarial sample generation method based on universal perturbation representation learning according to claim 4 is characterized in that: Step S4 specifically includes: Step S4-1, calculating the intra-modality loss based on the feature vectors of the anchor sample, positive sample, and negative sample of the image; Step S4-2, calculating the inter-modality loss using the anchor sample and positive sample of the image and the negative sample of the text; Step S4-3, obtaining the multimodal loss from the intra-modal loss and the inter-modal loss; Step S4-4: Update the adversarial perturbation using the multimodal loss.
6. The multimodal adversarial sample generation method based on universal perturbation representation learning according to claim 5 is characterized in that: The calculation expression of the intra-modal loss is: in, represents the intra-modal loss, is the InfoNCE loss function, represents the adversarial sample as the anchor sample, represents the adversarial sample obtained after random cropping as a positive sample, x v +x ′ v Construct a clean image as a negative sample and its randomly data-augmented image; Among them, q is the value of the anchor sample corresponding to the input, k + is the value of the positive sample corresponding to the input, k - is the value of the negative sample, τ is the temperature coefficient, exp(·) represents the calculation of the natural exponential, i is the negative sample index, Indicates the value of the input corresponding to the i-th negative sample; The calculation expression of the inter-modal loss is: Among them, F t (·) represents the feature vector representation of the text output of the text encoder of the alternative model, x t Represents clean text as negative sample; The calculation expression of the multimodal loss is: in, represents the multimodal loss, δ v represents adversarial perturbation, ∈ v represents the size limit of the adversarial perturbation, and α is the control inter-modal loss The hyperparameter of weight, st‖δ v ‖ ∞ ≤∈ v Indicates that the anti-perturbation δ v The absolute value of all dimensions of does not exceed ∈ v ; The calculation expression for updating the adversarial disturbance by the multimodal loss is: in, Represents a clean image superimposed with universal adversarial perturbation δ v The adversarial examples obtained are Represents an alternative model about gradient.
7. The multimodal adversarial sample generation method based on universal perturbation representation learning according to claim 6, characterized in that: The multimodal dataset is the Flickr30k test set; the alternative model is the Contrastive Language-Image Pre-training model; the initialization loss function J(·) is the KL divergence loss function; the perturbation size limit ∈ v Set to 12 / 255; controls intermodal loss The hyperparameter α of the weight is set to 3.
8. A multimodal adversarial sample generation system based on universal perturbation representation learning, characterized by: include: A determination module, configured to determine a multimodal dataset and a replacement model, wherein the multimodal dataset includes a plurality of samples consisting of images and texts; An initialization module, configured to perform a perturbation initialization process based on images in samples in a multimodal dataset and the substitution model to obtain an initial value for the adversarial perturbation; An extraction module is configured to collect a sample from a multimodal dataset, construct the sample based on the adversarial perturbation, perform feature extraction based on the substitution model, and obtain feature vectors of anchor samples, positive samples, and negative samples of the image in the sample and feature vectors of negative samples of the text in the sample; a multimodal learning module for performing intra-modal and inter-modal comparative learning based on feature vectors of anchor samples, positive samples, and negative samples of images and feature vectors of negative samples of text, obtaining a multimodal loss, updating the adversarial perturbation based on the multimodal loss, and finally outputting the adversarial perturbation; The superposition module is used to superimpose the adversarial perturbation on any image in the sample to obtain an adversarial sample.