A Multimodal Universal Adversarial Perturbation Method for Vision-Language Models

Abstract

The present invention relates to the technical field of computer systems based on specific computing models, and in particular to a multimodal universal adversarial perturbation method for a visual language model, comprising the following steps: obtaining a data set and a visual language model; encoding text samples and image samples in a training set and a verification set using the visual language model; initializing a multimodal universal adversarial perturbation, and optimizing the multimodal universal adversarial perturbation; judging whether a replacement update strategy is required; attacking samples in a training set and a verification set, and using samples that fail to attack as the training set and verification set of the next pair of multimodal universal adversarial perturbations; iteratively training the multimodal universal adversarial perturbation until the multimodal universal adversarial perturbation attack succeeds or all attacks fail, and recording the input samples that succeed and fail in the test set. The method provided by the present invention solves the problem that the attack effect of a unimodal universal adversarial perturbation on a visual language model is insufficient.

Description

Technical Field

[0001] The present invention relates to the technical field of computer systems based on specific computing models, and in particular to a multimodal universal adversarial perturbation method for a visual language model. Background Art

[0002] Visual language models have made great progress and significantly improved the performance in a range of multimodal downstream tasks. However, recent studies have shown that visual language models are still vulnerable to adversarial attacks. Unlike unimodal adversarial attacks that target a single modality, attacks on visual language models introduce perturbations in both image and text modalities, which increases the complexity of the interaction between different modalities.

[0003] Existing adversarial attack methods for visual language models target specific inputs and generate perturbations for each sample in the dataset. Therefore, each perturbation is tailored to fool the visual language model of a specific sample, but such perturbations often cannot be generalized to other samples.

[0004] There is no research on multimodal universal adversarial perturbations for visual language models, although universal adversarial perturbation techniques have been widely studied in unimodal settings and have proven their superiority. Universal adversarial perturbations are independent of the input data and can therefore be used on any sample in the dataset to deceive the model. These perturbations can be easily propagated to the Internet and widely used in attacks, which poses a huge threat and security risk to the deployment of models in real-world scenarios. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a multimodal universal adversarial perturbation method for a visual language model, which solves the problem that a unimodal universal adversarial perturbation has insufficient attack effect on a multimodal visual language model.

[0006] The present invention is achieved through the following technical solutions:

[0007] A multimodal universal adversarial perturbation method for a visual language model comprises the following steps:

[0008] S1: Obtain the dataset and visual language model, and divide the dataset into training set, validation set and test set;

[0009] S2: Use the visual language model to encode the text samples and image samples in the training set and validation set respectively;

[0010] S3: Initialize the multimodal universal adversarial perturbation and optimize the multimodal universal adversarial perturbation based on the encoding of text samples and image samples in the training set;

[0011] S4: Detect the performance of the optimized multimodal universal adversarial perturbation through the validation set, retain the best perturbation, and determine whether the update strategy needs to be replaced. If the update strategy does not need to be changed, keep the current update strategy to update the multimodal universal adversarial perturbation. If the update strategy needs to be changed, replace the update strategy and update the multimodal universal adversarial perturbation.

[0012] S5: Use the updated multimodal universal adversarial perturbation to attack the samples in the training set and the validation set respectively, and use the samples that fail the attack as the training set and validation set of the next pair of multimodal universal adversarial perturbations respectively;

[0013] S6: Iteratively train the multimodal universal adversarial perturbation, put the multimodal universal adversarial perturbation formed in each iteration into the attack queue, and attack the input samples in the test set in the order of the multimodal universal adversarial perturbations in the attack queue until the multimodal universal adversarial perturbations in the attack queue succeed or all attacks fail, and record the input samples that succeed and fail in the test set respectively.

[0014] Optimized, the dataset in step S1 is the MSCOCO dataset, which contains pictures from multiple scenes and their corresponding textual descriptive captions.

[0015] Optimized, at least 5 descriptive captions corresponding to the pictures of each scene.

[0016] Optimized, the visual language model in step S2 includes an image encoder and a text encoder, the image encoder is used to encode image samples in the data set, and the text encoder is used to encode text samples in the data set.

[0017] Furthermore, when initializing the multimodal universal adversarial perturbation in step S3, the universal adversarial perturbation of the image modality is initialized using a 0 matrix with the same shape as the input data, and the universal adversarial perturbation of the text modality is initialized using the letter A.

[0018] Furthermore, in step S3, the method for optimizing the multimodal universal adversarial perturbation based on the encoding of the text samples and image samples in the training set is as follows:

[0019] Optimization of Generic Adversarial Perturbations in Image Modality:

[0020] S311: Input the universal adversarial perturbation of the image modality into the visual language model, and use the Adam optimizer to optimize the loss function to update the universal adversarial perturbation of the image modality;

[0021] S312: After updating the universal adversarial perturbation of the image modality, constrain the universal adversarial perturbation of the image modality according to formula (1):

[0022] (1);

[0023] in: represents the universal adversarial perturbation of the image modality, represents the infinite norm of the universal adversarial perturbation of the image modality, represents the maximum disturbance amplitude for the image, Symbols that indicate limiting conditions, It means constraining the universal adversarial perturbation of the image modality to the value of the maximum perturbation amplitude with infinite norm;

[0024] Optimization of Generic Adversarial Perturbations in Text Modality:

[0025] S321: Use the embedding layer in the text modality encoding process of the visual language model as the initialization vocabulary;

[0026] S322: Adopting the momentum-based update method to optimize and update the universal adversarial perturbation of the text modality according to formula (2):

[0027] (2);

[0028] in: express Momentum that is updated all the time, express Momentum that is updated all the time, represents the decay rate of momentum, express The vector value of the perturbation word changes at time, represents the loss function, represents the learning rate, represents the vector of the perturbation word during the update process, express The vector of the perturbation word at time, express The disturbance words that are updated all the time, express The disturbance words that are updated all the time, It means to find the word with the highest similarity in the vocabulary according to the vector of the current perturbation word. Indicates a disturbing word, Represents a word in the vocabulary, represents all the words in the vocabulary, Represents a vocabulary, It means to calculate the cosine similarity of all words in the vocabulary. Indicates seeking similarity for the contents in brackets.

[0029] Optimized, the update strategies in step S4 include two types, one is a synchronous update strategy, for each batch of training data, only the universal adversarial perturbation of one modality is updated during training, and the universal adversarial perturbations of different modalities are updated alternately in different batches; the other is an asynchronous update strategy, for each batch of training data, the universal adversarial perturbations of two modalities are updated simultaneously during training.

[0030] Further, the method for determining whether the update strategy needs to be replaced in step S4 is as follows:

[0031] S41: Testing the performance of optimized multimodal universal adversarial perturbations through validation sets;

[0032] S42: Compare the performance of the optimized multimodal universal adversarial perturbation with the retained best perturbation. If the performance of the optimized multimodal universal adversarial perturbation exceeds the retained best perturbation, retain the optimized multimodal universal adversarial perturbation and keep the current update strategy to update the multimodal universal adversarial perturbation. If the performance of the optimized multimodal universal adversarial perturbation does not exceed the retained best perturbation, replace the update strategy and update the multimodal universal adversarial perturbation.

[0033] Optimized, in step S41, the performance of the optimized multimodal universal adversarial perturbation is detected by testing the success rate of the multimodal universal adversarial perturbation attack on the validation set.

[0034] Furthermore, when the update strategy is a synchronous update strategy and only the image modality is updated, the synchronous update loss function is calculated according to formula (3); when the update strategy is an asynchronous update strategy, the asynchronous update loss function is calculated according to formula (4), and the multimodal universal adversarial perturbation is updated according to the synchronous update loss function or the asynchronous update loss function:

[0035]

[0036] in: represents the synchronous update loss function, Represents a sample image. represents the universal adversarial perturbation of the image modality, represents the best historical text modality universal adversarial perturbation in the current training process, represents a text sample, Indicates that the contents in the brackets are trimmed. Indicates the use of an image encoder to encode the perturbed image after the image value is normalized to the normal range by the cropping operation. represents the encoding of the text sample after adding the historical optimal text perturbation, It means to find the similarity of the contents in brackets. It means to calculate the cosine similarity of the contents in brackets. represents the asynchronous update loss function, represents the perturbed image sample, represents the perturbed text sample, represents the perturbed image sample encoding, Represents the perturbed text sample encoding.

[0037] Beneficial effects of the invention:

[0038] The present invention provides a multimodal universal adversarial perturbation method for a visual language model. The performance of the optimized multimodal universal adversarial perturbation is detected by a validation set to adjust and optimize the update strategy of the universal adversarial perturbation, so that the universal adversarial perturbations of different modalities cooperate better and the attack effect is more obvious. In addition, through multiple rounds of universal adversarial perturbation attacks and using samples that failed in the previous attack for training, the multimodal universal adversarial perturbations trained in multiple rounds can well avoid the phenomenon of overfitting some samples, and the attack effect of the multimodal universal adversarial perturbation has been significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0040] A multimodal universal adversarial perturbation method for a visual language model includes the following steps, and its flowchart is as follows: Figure 1 As shown:

[0041] S1: Obtain the dataset and visual language model, and divide the dataset into training set, validation set and test set;

[0042] The data set here is preferably the MSCOCO data set, which includes pictures from multiple scenes and their corresponding textual descriptive captions, and there are at least 5 textual descriptive captions corresponding to the pictures of each scene.

[0043] While ensuring that the test set remains unchanged, 10,000 samples can be randomly selected from the training set of the dataset multiple times, of which 7,500 samples are used as training samples and 2,500 samples are used as validation set samples. Through multiple random selections, the average attack performance of the multimodal universal adversarial perturbation in the entire test set is calculated.

[0044] The visual language model can be used as the target model for attack. Specifically, this method uses CLIP-CNN-RN101 and CLIP-ViT-B-32 in the visual language model CLIP as the target attack model, and uses R@1, R@5, and R@10 in the image retrieval task and text retrieval task as observation indicators for checking the model attack effect.

[0045] Here, CLIP-CNN-RN101 and CLIP-ViT-B-32 are two commonly used models in visual language models. R@K (Recall at K) is an evaluation indicator used to measure the ability of a retrieval system to find relevant items in the top K results. R@1, R@5, and R@10 respectively mean that at least one sample among the top 1, 5, and 10 retrieved samples must be a relevant matching sample.

[0046] S2: Use the visual language model to encode the text samples and image samples in the training set and validation set respectively;

[0047] The visual language model includes an image encoder and a text encoder. The image encoder is used to encode image samples in the data set, and the text encoder is used to encode text samples in the data set.

[0048] The encoding of the image modality of the visual language model is divided into using the resnet image encoder and using the ViT image encoder. The encoding of the image modality is expressed as , the encoding of the text mode is represented as .

[0049] S3: Initialize the multimodal universal adversarial perturbation and optimize the multimodal universal adversarial perturbation based on the encoding of text samples and image samples in the training set;

[0050] Specifically, when initializing the multimodal universal adversarial perturbation, the universal adversarial perturbation for the image modality can be initialized using a 0 matrix with the same shape as the input data, and the universal adversarial perturbation for the text modality can be initialized using the letter A.

[0051] Since the universal adversarial perturbation for the image modality is a tensor in the form of data, it can be directly optimized using the Adam optimizer.

[0052] Specifically, the method for optimizing the multimodal universal adversarial perturbation based on the encoding of text samples and image samples in the training set is as follows:

[0053] Optimization of Generic Adversarial Perturbations in Image Modality:

[0054] S311: Input the universal adversarial perturbation of the image modality into the visual language model, and use the Adam optimizer to optimize the loss function to update the universal adversarial perturbation of the image modality;

[0055] S312: After updating the universal adversarial perturbation of the image modality, constrain the universal adversarial perturbation of the image modality according to formula (1) so that the perturbation amplitude is guaranteed to be within a set range and the perturbation is as imperceptible to the human eye as possible.

[0056] (1);

[0057] in: represents the universal adversarial perturbation of the image modality, represents the infinite norm of the universal adversarial perturbation of the image modality, represents the maximum disturbance amplitude for the image, Symbols that indicate limiting conditions, It means constraining the universal adversarial perturbation of the image modality to the value of the maximum perturbation amplitude with infinite norm;

[0058] For the universal adversarial perturbation of the text modality, since its encoding is too discrete, this method uses the embedding layer in the text modality encoding process of the attacked model as the initialization vocabulary. In addition, in order to achieve faster and more stable convergence, a momentum-based update method is used to optimize the universal adversarial perturbation of the text modality. The specific process is as follows:

[0059] S321: Use the embedding layer in the text modality encoding process of the visual language model as the initialization vocabulary;

[0060] S322: Adopting the momentum-based update method to optimize and update the universal adversarial perturbation of the text modality according to formula (2):

[0061] (2);

[0062] in: express Momentum that is updated all the time, express Momentum that is updated all the time, represents the decay rate of momentum, express The vector value of the perturbation word changes at time, represents the loss function, represents the learning rate, represents the vector of the perturbation word during the update process, express The vector of the perturbation word at time, express The disturbance words that are updated all the time, express The disturbance words that are updated all the time, It means to find the word with the highest similarity in the vocabulary according to the vector of the current perturbation word. Indicates a disturbing word, Represents a word in the vocabulary, represents all the words in the vocabulary, Represents a vocabulary, It means to calculate the cosine similarity of all words in the vocabulary. Indicates seeking similarity for the contents in brackets.

[0063] S4: Detect the performance of the optimized multimodal universal adversarial perturbation through the validation set, retain the best perturbation, and determine whether the update strategy needs to be replaced. If the update strategy does not need to be changed, keep the current update strategy to update the multimodal universal adversarial perturbation. If the update strategy needs to be changed, replace the update strategy and update the multimodal universal adversarial perturbation.

[0064] Specifically, there are two update strategies. One is the synchronous update strategy. For each batch of training data, only the universal adversarial perturbation of one modality is updated during training, and the universal adversarial perturbations of different modalities are updated alternately in different batches. The other is the asynchronous update strategy. For each batch of training data, the universal adversarial perturbations of two modalities are updated simultaneously during training.

[0065] Specifically, the method for determining whether the update strategy needs to be replaced is as follows:

[0066] S41: Testing the performance of optimized multimodal universal adversarial perturbations through validation sets;

[0067] Specifically, the performance of the optimized multimodal universal adversarial perturbation can be detected by testing the success rate of the multimodal universal adversarial perturbation attack on the validation set.

[0068] S42: Compare the performance of the optimized multimodal universal adversarial perturbation with the retained best perturbation. If the performance of the optimized multimodal universal adversarial perturbation exceeds the retained best perturbation, retain the optimized multimodal universal adversarial perturbation and keep the current update strategy to update the multimodal universal adversarial perturbation. If the performance of the optimized multimodal universal adversarial perturbation does not exceed the retained best perturbation, replace the update strategy and update the multimodal universal adversarial perturbation.

[0069] The synchronous update strategy and the asynchronous update strategy each have different advantages and disadvantages. The asynchronous update strategy will make the update more rapid and extensive in the optimization direction of the better universal adversarial perturbation, which helps to jump out of the local optimal solution and quickly find a better universal adversarial perturbation solution in the early stage. However, in the asynchronous update process, the degree of coordination and different update amplitudes of the two modal perturbations of image and text will be largely ignored. The synchronous update strategy will make the update amplitudes of the universal adversarial perturbations of different modalities more matched, which helps to strengthen the coordination of the two modal perturbations to achieve better attack coordination effects under the current universal adversarial perturbations of each individual modality, but this method may make it difficult for the optimization to jump out of the local optimal solution.

[0070] Since both the separate asynchronous update and synchronous update strategies have their own shortcomings, although the asynchronous update can update the universal adversarial perturbations of the two modalities at the same time, since the universal perturbation of the text modality exists in the form of perturbation words, the overly discrete text perturbations will not be able to be timely optimized to the state that best matches the current text modality universal adversarial perturbation after frequent updates. When the text modality is updated from one perturbation word to another, it only requires the update of several batches of samples, while the image modality perturbation is difficult to be optimized to the best matching universal image perturbation for the new perturbation word during this period.

[0071] Although the synchronous update solves the problem of mismatch between the update speed of the image modality and the update speed of the text modality, and enables better coordination between the perturbation of the image modality and the perturbation of the text modality, it only updates one of the modalities during the update, resulting in a decrease in its performance in finding a better perturbation direction.

[0072] Therefore, this method uses the validation set to test the current multimodal universal adversarial perturbation attack effect to select the current appropriate update strategy. At the end of each training stage, the validation set is used to evaluate the current best multimodal universal adversarial perturbation effect and compare it with the previous best effect. If it is found that the attack effect of this stage exceeds the previous best record, the historical record is updated and the multimodal universal adversarial perturbation of this stage is retained. On the contrary, if the effect of this stage fails to surpass the historical best, this method will switch to a different optimization update strategy. Thereby making up for the shortcomings of the two update strategies, the universal adversarial perturbations of different modes cooperate better and the attack effect is more obvious.

[0073] Furthermore, when the update strategy is a synchronous update strategy and only the image modality is updated, the synchronous update loss function is calculated according to formula (3). When the update strategy is a synchronous update strategy and only the text modality is updated, the synchronous update loss function is calculated in the same way as formula (3). When the update strategy is an asynchronous update strategy, the asynchronous update loss function is calculated according to formula (4), and the multimodal universal adversarial perturbation is updated according to the synchronous update loss function or the asynchronous update loss function:

[0074]

[0075] in: represents the synchronous update loss function, Represents a sample image. represents the universal adversarial perturbation of the image modality, represents the best historical text modality universal adversarial perturbation in the current training process, represents a text sample, Indicates that the contents in the brackets are trimmed. Indicates the use of an image encoder to encode the perturbed image after the image value is normalized to the normal range by the cropping operation. represents the encoding of the text sample after adding the historical optimal text perturbation, It means to find the similarity of the contents in brackets. It means to calculate the cosine similarity of the contents in brackets. represents the asynchronous update loss function, represents the perturbed image sample, represents the perturbed text sample, represents the perturbed image sample encoding, Represents the perturbed text sample encoding.

[0076] Since the image-text retrieval task depends on the similarity between the image and the text, the above method is used to calculate the asynchronous update loss function and the synchronous update loss function respectively, and the multimodal universal adversarial perturbation is updated according to the synchronous update loss function or the asynchronous update loss function. This can destroy the similarity, that is, increase the difference, so that the multimodal universal adversarial perturbation attack is successful, thereby improving the accuracy of the multimodal universal adversarial perturbation performance judgment.

[0077] S5: Use the updated multimodal universal adversarial perturbation to attack the samples in the training set and the validation set respectively, and use the samples that fail the attack as the training set and validation set of the next pair of multimodal universal adversarial perturbations respectively;

[0078] Compared with the general adversarial perturbation of a single modality, the representation learning between the general adversarial perturbations of multimodal is more complicated. This complexity often leads to many failed attacks. For example, in the semantic embedding space, the general adversarial perturbation of the image modality can make it deviate from the original area, thereby destroying the alignment between the image and text modalities to achieve the purpose of successful attack. However, at this time, the general adversarial perturbation of the text modality moves the text features to the same area as the image, resulting in the realignment of the image and text features, which ultimately leads to the failure of the attack. In addition, the above situation is obviously successful only when the general adversarial perturbation of the image modality and the general adversarial perturbation of the text modality are in different directions. This shows that the general adversarial perturbation of a single modality cannot meet the cancellation of multimodal scenarios, because it can only adapt to the single offset direction of the original image and text pair. Therefore, this method adopts a multi-round general adversarial attack method, and iteratively uses the samples that failed in the previous attack to train and verify the new general adversarial perturbation to achieve the purpose of covering a wider range of attack directions.

[0079] S6: Iteratively train the multimodal universal adversarial perturbation, put the multimodal universal adversarial perturbation formed in each iteration into the attack queue, and attack the input samples in the test set in the order of the multimodal universal adversarial perturbations in the attack queue until the multimodal universal adversarial perturbations in the attack queue succeed or all attacks fail, and record the input samples that succeed and fail in the test set respectively.

[0080] The main indicators for evaluating the attack effect in this method are IR@1, IR@5, IR@10 in image retrieval tasks and TR@1, TR@5, TR@10 in text retrieval tasks. IR means retrieving text through images, and TR means retrieving images through text.

[0081] This method is experimented on the MSCOCO dataset. Under the premise of ensuring that the test set remains unchanged, 12,500 samples are randomly extracted from the original training set multiple times, of which 10,000 samples are used for training and 2,500 samples are used for verification. The effectiveness of this method is verified by taking the average results of multiple experiments.

[0082] The universal adversarial perturbation method proposed in the present invention has two types of update strategies, namely synchronous update strategy and asynchronous update strategy. The comparison methods used as the control group are UAT (universal adversarial perturbation in text mode), UAP (universal adversarial perturbation in image mode), UAP+UAT (hybrid adversarial perturbation), UAP+UAT (multi-round attack hybrid adversarial perturbation), M^{3}UAP-HAS (multi-mode, multi-round universal adversarial perturbation with synchronous-asynchronous hybrid update strategy). The results of attacking the CLIP-CNN model and the CLIP-ViT model are shown in Tables 1 and 2:

[0083] Table 1

[0084]

[0085] Table 2

[0086]

[0087] The reported values ​​in Table 1 and Table 2 are the matching success rates, and the lower the rate, the better the attack effect. By comparing the 4th row with the 2nd and 3rd rows of the two tables, it can be found that the multimodal universal adversarial perturbation has further improved the attack effect compared with the unimodal universal adversarial perturbation. This shows the feasibility of the universal adversarial perturbation attack. Secondly, comparing the single-round attack and the double-round attack in rows 4, 5, and 6, the double-round attack has significantly improved the attack effect compared with the single-round attack. Finally, comparing the results of rows 6 and 7 with row 4, it is found that it is more important to consider the interaction and collaborative optimization between modalities in each update, which proves the effectiveness of the hybrid update strategy of this method.

[0088] In order to further explore the importance of different attack strategies in the general adversarial perturbation attack method, ablation experiments using only asynchronous update strategy, only synchronous update strategy and hybrid update strategy were designed. The experimental results are shown in Table 3:

[0089] Table 3

[0090]

[0091] Comparing the second row with the first row in Table 3, we find that the asynchronous update strategy is always better than the synchronous update strategy. Secondly, comparing the third row with rows 1 and 2, we find that the hybrid update strategy achieves the best results in all indicators. This fully demonstrates that combining the synchronous update strategy and the asynchronous update strategy can better play their respective advantages.

[0092] In summary, the present invention provides a multimodal universal adversarial perturbation method for a visual language model. The performance of the optimized multimodal universal adversarial perturbation is detected by a verification set to adjust and optimize the update strategy of the universal adversarial perturbation, so that the universal adversarial perturbations of different modalities can cooperate better and the attack effect is more obvious. Through multiple rounds of universal adversarial perturbation attacks, samples that failed in the previous attack are used for training, so that the multimodal universal adversarial perturbations trained through multiple rounds can well avoid the phenomenon of overfitting some samples, and the attack effect of the multimodal universal adversarial perturbation is significantly improved.

[0093] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multimodal universal adversarial perturbation method for a visual language model, characterized by: The steps include: S1: Obtain the dataset and visual language model, and divide the dataset into training set, validation set and test set; S2: Use the visual language model to encode the text samples and image samples in the training set and validation set respectively; S3: Initialize the multimodal universal adversarial perturbation and optimize the multimodal universal adversarial perturbation based on the encoding of text samples and image samples in the training set; S4: Detect the performance of the optimized multimodal universal adversarial perturbation through the validation set, retain the best perturbation, and determine whether the update strategy needs to be replaced. If the update strategy does not need to be changed, keep the current update strategy to update the multimodal universal adversarial perturbation. If the update strategy needs to be changed, replace the update strategy and update the multimodal universal adversarial perturbation. S5: Use the updated multimodal universal adversarial perturbation to attack the samples in the training set and the validation set respectively, and use the samples that fail the attack as the training set and validation set of the next pair of multimodal universal adversarial perturbations respectively; S6: Iteratively train the multimodal universal adversarial perturbation, put the multimodal universal adversarial perturbation formed in each iteration into the attack queue, and attack the input samples in the test set in the order of the multimodal universal adversarial perturbations in the attack queue until the multimodal universal adversarial perturbations in the attack queue succeed or all attacks fail, and record the input samples that succeed and fail in the test set respectively.

2. The multimodal universal adversarial perturbation method for a visual language model according to claim 1, characterized in that: The dataset in step S1 is the MSCOCO dataset, which includes pictures from multiple scenes and their corresponding textual descriptive captions.

3. The multimodal universal adversarial perturbation method for a visual language model according to claim 2, characterized in that: There should be at least 5 descriptive captions corresponding to the pictures of each scene.

4. The multimodal universal adversarial perturbation method for a visual language model according to claim 1, characterized in that: In step S2, the visual language model includes an image encoder and a text encoder. The image encoder is used to encode image samples in the data set, and the text encoder is used to encode text samples in the data set.

5. The multimodal universal adversarial perturbation method for a visual language model according to claim 1, characterized in that: When initializing the multimodal universal adversarial perturbation in step S3, the universal adversarial perturbation of the image modality is initialized using a 0 matrix with the same shape as the input data, and the universal adversarial perturbation of the text modality is initialized using the letter A.

6. The multimodal universal adversarial perturbation method for a visual language model according to claim 5, characterized in that: The method for optimizing the multimodal universal adversarial perturbation based on the encoding of the text samples and image samples in the training set in step S3 is as follows: Optimization of Generic Adversarial Perturbations in Image Modality: S311: Input the universal adversarial perturbation of the image modality into the visual language model, and use the Adam optimizer to optimize the loss function to update the universal adversarial perturbation of the image modality; S312: After updating the universal adversarial perturbation of the image modality, constrain the universal adversarial perturbation of the image modality according to formula (1): （1）； in: represents the universal adversarial perturbation of the image modality, represents the infinite norm of the universal adversarial perturbation of the image modality, represents the maximum disturbance amplitude for the image, Symbols that indicate limiting conditions, It means constraining the universal adversarial perturbation of the image modality to the value of the maximum perturbation amplitude with infinite norm; Optimization of Generic Adversarial Perturbations in Text Modality: S321: Use the embedding layer in the text modality encoding process of the visual language model as the initialization vocabulary; S322: Adopting the momentum-based update method to optimize and update the universal adversarial perturbation of the text modality according to formula (2): （2）； in: express Momentum that is updated all the time, express Momentum that is updated all the time, represents the decay rate of momentum, express The vector value of the perturbation word changes at time, represents the loss function, represents the learning rate, represents the vector of the perturbation word during the update process, express The vector of the perturbation word at time, express The disturbance words that are updated all the time, express The disturbance words that are updated all the time, It means to find the word with the highest similarity in the vocabulary according to the vector of the current perturbation word. Indicates a disturbing word, Represents a word in the vocabulary, represents all the words in the vocabulary, Represents a vocabulary, It means to find the similarity of the contents in brackets. Indicates that the cosine similarity of the contents in the brackets is calculated.

7. The multimodal universal adversarial perturbation method for a visual language model according to claim 1, characterized in that: There are two types of update strategies in step S4. One is a synchronous update strategy. For each batch of training data, only the universal adversarial perturbation of one modality is updated during training, and universal adversarial perturbations of different modalities are updated alternately in different batches. The other is an asynchronous update strategy, where for each batch of training data, the universal adversarial perturbations of the two modalities are updated simultaneously during training.

8. The multimodal universal adversarial perturbation method for a visual language model according to claim 1, characterized in that: The method for determining whether the update strategy needs to be replaced in step S4 is as follows: S41: Testing the performance of optimized multimodal universal adversarial perturbations through validation sets; S42: Compare the performance of the optimized multimodal universal adversarial perturbation with the retained best perturbation. If the performance of the optimized multimodal universal adversarial perturbation exceeds the retained best perturbation, retain the optimized multimodal universal adversarial perturbation and keep the current update strategy to update the multimodal universal adversarial perturbation. If the performance of the optimized multimodal universal adversarial perturbation does not exceed the retained best perturbation, replace the update strategy and update the multimodal universal adversarial perturbation.

9. The multimodal universal adversarial perturbation method for a visual language model according to claim 1, characterized in that: In step S41, the performance of the optimized multimodal universal adversarial perturbation is detected by testing the success rate of the multimodal universal adversarial perturbation attack on the validation set.

10. The multimodal universal adversarial perturbation method for a visual language model according to claim 7, characterized in that: In step S4, when the update strategy is a synchronous update strategy and only the image modality is updated, the synchronous update loss function is calculated according to formula (3); when the update strategy is an asynchronous update strategy, the asynchronous update loss function is calculated according to formula (4), and the multimodal universal adversarial perturbation is updated according to the synchronous update loss function or the asynchronous update loss function: in: represents the synchronous update loss function, Represents a sample image. represents the universal adversarial perturbation of the image modality, represents the best historical text modality universal adversarial perturbation in the current training process, represents a text sample, Indicates that the contents in the brackets are trimmed. Indicates the use of an image encoder to encode the perturbed image after the image values ​​are normalized to the normal range by the cropping operation. represents the encoding of the text sample after adding the historical optimal text perturbation, It means to find the similarity of the contents in brackets. It means to calculate the cosine similarity of the contents in brackets. represents the asynchronous update loss function, represents the perturbed image sample, represents the perturbed text sample, represents the perturbed image sample encoding, Represents the perturbed text sample encoding.

Images