Cross-modal transferable backdoor attack method and device

Abstract

The invention relates to the technical field of computer vision and natural language processing, and provides a cross-modal transferable backdoor attack method and device, and the method comprises the steps: constructing a backdoor data set containing a trigger based on an unlabeled original data set; inputting the original data set and the backdoor data set containing the trigger into a pre-constructed hacker network, and respectively calculating clean data silence loss and backdoor toxicity loss with the trigger; pre-training a pre-constructed hacker network by minimizing clean data silence loss and backdoor toxicity loss; calculating the feature representation of the target category through the image data and the text data of the target category, and performing backdoor transfer by using the pre-trained hacker network and the feature representation of the target category to generate a target CLIP model with a backdoor; and attacking by using the generated model. The backdoor trigger can be embedded in multi-modal data at the same time, and the method has high cross-modal transferability and concealment and does not depend on large-scale annotation data.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and natural language processing, and in particular to a cross-modal transferable backdoor attack method and device. Background Art

[0002] With the rapid development of deep learning technology, cross-modal learning has been widely used in a variety of artificial intelligence tasks, such as image recognition, natural language processing, and multimodal reasoning. As a representative cross-modal learning method, the Contrastive Language-Image Pre-training (CLIP) model has successfully mapped image and text information into a shared semantic space and achieved remarkable results in multiple tasks. However, with the popularity of the model, its security issues have gradually been exposed, especially in the face of malicious attacks, the robustness of the model is particularly weak.

[0003] In recent years, backdoor attacks have attracted widespread attention in deep learning models as a potential security threat. Backdoor attacks inject specific triggers during the training process, making the model perform well under normal circumstances, but produce incorrect prediction results when subjected to specific inputs (i.e., samples with triggers). Although most existing research focuses on backdoor attacks in single modalities such as images or text, with the rise of multimodal models, single-modal attack methods can no longer meet the attack requirements for cross-modal models. Therefore, how to simultaneously inject backdoors into different modal data such as images and text has become a challenge that needs to be solved urgently.

[0004] Most existing cross-modal backdoor attack methods rely on image-text paired data, lacking in concealment and cross-modal transferability. Summary of the invention

[0005] The present invention provides a cross-modal transferable backdoor attack method and device, which is used to solve the defects that most existing cross-modal backdoor attack methods rely on image-text paired data, lack of concealment and cross-modal transferability, and realize a cross-modal backdoor attack with strong concealment and significant effect. The technical solution proposed by the present invention is as follows: In a first aspect, the present invention provides a cross-modal transferable backdoor attack method, comprising: Obtain an unlabeled original data set, and construct a backdoor data set containing a trigger based on the original data set; Input the original data set and the backdoor data set containing the trigger into a pre-built hacker network, and calculate the clean data silence loss and the backdoor toxicity loss with the trigger respectively; wherein the clean data silence loss corresponds to the original data set, and the backdoor toxicity loss with the trigger corresponds to the backdoor data set containing the trigger; Pre-training the pre-built hacker network by minimizing the clean data silence loss and the backdoor toxicity loss with triggers to obtain a pre-trained hacker network; Calculate the feature representation of the target category through the image data and text data of the target category, perform backdoor transfer using the pre-trained hacker network and the feature representation of the target category, and generate a target CLIP model with a backdoor; The attack is performed using the target CLIP model with the backdoor.

[0006] Optionally, the original data set includes an image data set and a text data set, and the backdoor data set includes an image data set containing a trigger and a text data set containing a trigger; The step of constructing a backdoor data set containing a trigger based on the original data set includes: Applying the image trigger function to each image in the image dataset to generate an image dataset containing the trigger; Apply the text trigger function to each text in the text dataset to generate a text dataset containing the trigger.

[0007] Optionally, the characteristic value output by the hacker network is zero when there is no trigger in the input data, and the characteristic value output by the hacker network is greater than a preset threshold when there is a trigger in the input data.

[0008] Optionally, the clean data silent loss includes a clean image silent loss and a clean text silent loss, the backdoor toxicity loss with trigger includes a backdoor image toxicity loss and a backdoor text toxicity loss, and the pre-built hacker network includes a visual hacker network and a text hacker network; The pre-training of the pre-built hacker network by minimizing the clean data silence loss and the backdoor toxicity loss with triggers to obtain the pre-trained hacker network includes: Calculating a first total loss based on the clean image silence loss and the backdoor image toxicity loss, and pre-training the visual hacker network based on the first total loss to obtain a pre-trained visual hacker network; A second total loss is calculated based on the clean text silence loss and the backdoor text toxicity loss, and the text hacker network is pre-trained based on the second total loss to obtain a pre-trained text hacker network.

[0009] Optionally, calculating the feature representation of the target category through the image data and text data of the target category includes: The feature representation of the target category is calculated based on the image data and text data of the target category by the following formula:

[0010] In the formula, is the feature representation of the target category, is the target category, is the normalization function, and From the target category The number of images and texts sampled in, and From the target category The image dataset and text dataset sampled from For the target category image data, For the target category The text data, and They are the image encoder and text encoder of the target CLIP model, Image encoder using the target CLIP model Processing target category Image data The obtained feature vector is For text encoders using the target CLIP model Processing target category Text data The resulting feature vector.

[0011] Optionally, the pre-trained hacker network includes a pre-trained visual hacker network and a pre-trained text hacker network; the step of performing backdoor transfer using the pre-trained hacker network and the feature representation of the target category to generate a target CLIP model with a backdoor includes: The output of the pre-trained hacker network and the feature representation of the target category are used to perform backdoor transfer through the following formula to generate a target CLIP model with a backdoor:

[0012] In the formula, For input data, and They are the image features and text features output by the target CLIP model with backdoor, is the normalization function, and They are the image encoder and text encoder of the target CLIP model, is the output of the pre-trained visual hacking network, is the output of the pre-trained text hacking network, is the feature representation of the target category.

[0013] In a second aspect, the present invention further provides a cross-modal transferable backdoor attack device, comprising the following modules: A data set construction module, used to obtain an unlabeled original data set, and to construct a backdoor data set containing a trigger based on the original data set; A loss calculation module, used for inputting the original data set and the backdoor data set containing triggers into a pre-built hacker network, and respectively calculating the clean data silence loss and the backdoor toxicity loss with triggers; wherein the clean data silence loss corresponds to the original data set, and the backdoor toxicity loss with triggers corresponds to the backdoor data set containing triggers; A pre-training module, used for pre-training the pre-built hacker network by minimizing the clean data silence loss and the backdoor toxicity loss with triggers to obtain a pre-trained hacker network; A backdoor transfer module, used to calculate the feature representation of the target category through the image data and text data of the target category, perform backdoor transfer using the pre-trained hacker network and the feature representation of the target category, and generate a target CLIP model with a backdoor; The attack module is used to attack using the target CLIP model with the backdoor.

[0014] In a third aspect, the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, the cross-modal transferable backdoor attack method as described in the first aspect above is implemented.

[0015] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the cross-modal transferable backdoor attack method as described in the first aspect above.

[0016] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the cross-modal transferable backdoor attack method as described in the first aspect above.

[0017] Based on the above technical solution, the beneficial effects of the present invention compared with the prior art are as follows: The cross-modal transferable backdoor attack method and device provided by the present invention construct a backdoor dataset by embedding triggers in the original dataset. These triggers can be secret and difficult to detect, thereby enhancing the concealment of the attack. The hacker network is pre-trained by minimizing the silent loss of clean data and the toxic loss of the backdoor with triggers. This process helps the hacker network learn how to identify and respond to triggers without attracting attention, further enhancing the concealment of the attack. The method not only relies on image-text paired data, but also can enhance the cross-modal transferability of the attack through a wider range of cross-modal data (such as audio, video, etc.). The backdoor transfer is performed using the feature representation of the pre-trained hacker network and the target category to generate a target CLIP model with a backdoor. This process enables the backdoor attack to be transferred between different modalities, improving the flexibility and adaptability of the attack. Through the pre-training and backdoor transfer mechanism of the hacker network, the present invention can generate a target CLIP model with a backdoor, which can generate an output specified by the attacker when facing an input containing a trigger, while maintaining normal performance for clean samples. Compared with traditional cross-modal backdoor attack methods, this method achieves higher attack performance with less annotated data and stronger concealment.

[0018] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0019] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 It is a flow chart of the cross-modal transferable backdoor attack method provided by the present invention.

[0022] Figure 2It is a comparison diagram of the traditional backdoor attack method and the cross-modal transferable backdoor attack (TransferableBackdoor Attacks, TBA) method proposed in the present invention on the CLIP model; among them, (a) is the traditional backdoor attack method, and (b) is the cross-modal transferable backdoor attack method of the present invention.

[0023] Figure 3 It is a schematic diagram of the structure of the fused multimodal encoder (image encoder and text encoder) of the backdoor embedded CLIP model.

[0024] Figure 4 This is an overview of the multimodal hacker network pre-training.

[0025] Figure 5 An overview of backdoor transfer for few-shot training.

[0026] Figure 6 Qualitative results of transferable backdoor attacks on cross-modal retrieval on the MSCOCO dataset; (a) is the image retrieval text task, and (b) is the text retrieval image task.

[0027] Figure 7 It is a schematic diagram of the structure of the cross-modal transferable backdoor attack device provided by the present invention.

[0028] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] Combine the following Figure 1-Figure 7 The cross-modal transferable backdoor attack method and apparatus of the present invention are described.

[0031] Reference Figure 1 As shown in the figure, the cross-modal transferable backdoor attack method aims to effectively embed the backdoor in the CLIP model to achieve cross-modal attack capabilities. The method includes the following: Step S110: Acquire an unlabeled original data set, and construct a backdoor data set containing triggers based on the original data set.

[0032] Unlike the existing backdoor attack methods based on image-text pairs, the hacker network of the present invention only requires unlabeled unimodal data (i.e., image data or text data) for pre-training. A large amount of unlabeled image and text data is obtained from public data sources or internal databases to form an unlabeled original data set. In order to ensure the effectiveness and concealment of the backdoor attack, unique triggers are designed for images and texts respectively. An image trigger can be a small pattern or a specific color combination that is not easily noticeable under normal circumstances, but can activate the backdoor under certain conditions. A text trigger can be a specific phrase or word that may seem harmless in regular text but can trigger the backdoor mechanism when embedded in text.

[0033] After designing the triggers, we embed them into the original dataset to generate a backdoor dataset containing triggers. For images, we embed trigger patterns at specific locations of the image (such as corners, edges, etc.); for text data, we insert trigger phrases at specific locations of the text (such as the beginning, end, or in a specific paragraph).

[0034] Step S120, input the original data set and the backdoor data set containing the trigger into a pre-built hacker network, and calculate the clean data silence loss and the backdoor toxicity loss with the trigger respectively; wherein the clean data silence loss corresponds to the original data set, and the backdoor toxicity loss with the trigger corresponds to the backdoor data set containing the trigger.

[0035] Pre-build a hacker network that can process image and text data and output feature values. For images, you can use Convolutional Neural Networks (CNN), VisionTransformer (ViT), or Residual Network (ResNet) architectures; for text, you can use Recurrent Neural Networks (RNN), Long Short-Term Memory Networks (LSTM), or Transformer (such as BERT).

[0036] After preparing the pre-training data, the goal is to pre-train the visual hacker network and text hacker network Specifically, the original data set (i.e., clean data) and the backdoor data set containing triggers are prepared separately as inputs to the hacker network. For clean data, the clean data silence loss is calculated to ensure that the hacker network outputs a feature value close to 0 when inputting clean data, thereby maintaining the normal behavior of the network under normal circumstances. For the backdoor data set containing triggers, the backdoor toxicity loss with triggers is calculated to ensure that the hacker network outputs a feature value greater than a certain threshold when inputting data containing triggers, thereby activating the backdoor mechanism.

[0037] Step S130: pre-train the pre-built hacker network by minimizing the clean data silence loss and the backdoor toxicity loss with triggers to obtain a pre-trained hacker network.

[0038] When pre-training the hacker network, select a suitable optimizer, such as Adam, SGD, etc., to update the weights of the hacker network. The gradient of the loss is calculated through the back-propagation algorithm, and the optimizer is used to update the weights of the hacker network to minimize the clean data silent loss and the backdoor toxicity loss with triggers. This process needs to be repeated for multiple iterations until the hacker network reaches the predetermined performance standard on the validation set or the number of training rounds reaches the preset upper limit.

[0039] Step S140, calculating the feature representation of the target category through the image data and text data of the target category, performing backdoor transfer using the pre-trained hacker network and the feature representation of the target category, and generating a target CLIP model with a backdoor.

[0040] Under the target CLIP model and target category, a backdoor transfer step is performed. This step aims to efficiently embed the backdoor into the CLIP model with a small number of samples (i.e., a small amount of image data and text data of the target category) without further training. Specifically, first, the feature representation of the target category is calculated using the image data and text data of the target category. The feature representation of the target category is a typical representation of the target category in the feature space, and they can represent the core features of the target category. Then, a backdoor transfer operation is performed. This operation combines the output of the pre-trained hacker network with the feature representation of the target category, and embeds the backdoor into the target CLIP model in some way (such as weighted summation, splicing, etc.). For the image encoder and text encoder of the target CLIP model, the output of the hacker network is combined with the feature representation of the target category to generate a CLIP model with a backdoor.

[0041] Step S150: attack using the target CLIP model with the backdoor.

[0042] In the attack phase, image data and text data containing triggers are prepared as attack data. These attack data are similar to ordinary data in appearance, but the designed triggers are embedded. The attack data is input into the target CLIP model with backdoor. Observe whether the output of the model is biased towards the target category. If the output features of the model are similar or identical to the feature representation of the target category, it indicates that the backdoor attack is successful. This means that the model is induced to output results related to the target category when receiving data containing triggers. The effect of the backdoor attack is evaluated by comparing the model output before and after the attack. The success rate of the attack can be measured using indicators such as accuracy, recall, and F1 score. These indicators can intuitively reflect the degree of influence of the backdoor attack on the model behavior, thereby helping to evaluate the efficiency and concealment of the attack.

[0043] The above image trigger is a carefully designed pattern or object that, when added to an image, can trigger the model to produce a specific erroneous output. For example, in a face recognition system, an attacker can design a special glasses pattern as an image trigger. In the training data set, the attacker will add this glasses pattern to certain people's faces and mark these images with triggers as another person's identity. When the model is trained, it will mistake anyone wearing these special glasses as the person marked in the training set.

[0044] A text trigger is a specific text or character sequence that, when added to the input text, can also trigger the model's erroneous output. Taking the sentiment analysis model as an example, an attacker can design a specific phrase or word as a text trigger. In the training dataset, the attacker will add this phrase to some positively reviewed texts, but mark these texts as negative reviews. In this way, when the model is trained, it will mistake any positive review containing this trigger phrase as a negative review.

[0045] In a more complex scenario, an attacker may combine both image and text triggers to launch an attack. For example, in an image description generation system, an attacker can design a specific logo pattern (image trigger) and a specific text prompt (text trigger). In the training dataset, the attacker will add this logo pattern to certain images and add specific text prompts to the description text of these images. Then, these images with triggers and description texts are used together to train the model. When the model is trained, it will generate the attacker-specified incorrect description on any image that contains this logo pattern and specific text prompt. This attack method that combines image and text triggers can significantly improve the performance and portability of backdoor attacks because the attacker can use more features to trigger the model's incorrect output.

[0046] The cross-modal backdoor attack method provided by the present invention uses an unlabeled original data set to construct a backdoor data set. First, the unlabeled original data set is obtained, which reduces the tediousness of data preparation work, because in practical applications, labeled data is often costly and difficult to obtain. The backdoor data set containing triggers is constructed based on the original data set. This process does not require additional labeling information, thereby reducing the need for data labeling and overcoming the dependence on a large amount of labeled data.

[0047] The cross-modal backdoor attack method provided by the present invention can embed backdoor triggers in multiple modal data at the same time, has strong cross-modal transferability and concealment, and does not rely on large-scale annotated data. The method can accurately attack specific target categories without destroying the normal function of the model, and has broad application prospects. Specifically, the present invention constructs a backdoor dataset by embedding triggers in the original dataset. These triggers can be secret and difficult to detect, thereby enhancing the concealment of the attack. The hacker network is pre-trained by minimizing the silent loss of clean data and the toxicity loss of the backdoor with triggers. This process helps the hacker network learn how to identify and respond to triggers without attracting attention, further enhancing the concealment of the attack. The method not only relies on image-text paired data, but also can enhance the cross-modal transferability of the attack through a wider range of cross-modal data (such as audio, video, etc.). The backdoor transfer is performed using the feature representation of the pre-trained hacker network and the target category to generate a target CLIP model with a backdoor. This process enables the backdoor attack to be transferred between different modalities, improving the flexibility and adaptability of the attack. Through the pre-training of the hacker network and the backdoor transfer mechanism, the method can generate a target CLIP model with a backdoor, which can produce the output specified by the attacker when faced with an input containing a trigger, while maintaining normal performance for clean samples. Compared with traditional cross-modal backdoor attack methods, this method achieves higher attack performance with less annotated data and stronger concealment.

[0048] The above-mentioned hacker network in the present invention outputs a characteristic value of zero when there is no trigger in the input data, and outputs a characteristic value greater than a preset threshold when the input data contains a trigger.

[0049] The hacker network (i.e., backdoor attack model) is trained using a dataset containing triggers. During the training process, the model learns to recognize and respond to triggers. When there are no triggers in the input data, the model maintains normal behavior; when the input data contains triggers, the model produces a specific output (i.e., the feature value is greater than the preset threshold). After the model is deployed, the hacker network performs real-time detection on the input data. The feature value is calculated to determine whether the input data contains a trigger. If the feature value is zero, it indicates that there is no trigger in the input data and the network maintains normal behavior. If the feature value is greater than the preset threshold, it indicates that the input data contains a trigger and the network performs a backdoor attack.

[0050] The present invention can embed a backdoor in a hacker network without changing the main features of the original data by designing triggers and training models. This makes the backdoor attack highly concealed and difficult to detect or discover. When the input data contains a trigger, the hacker network can quickly identify and generate a specific output. This makes the backdoor attack have a high attack effect and can achieve the hacker's malicious purpose. Different types of triggers can be designed and different models can be trained as needed. This makes the backdoor attack highly flexible and can adapt to different attack scenarios and targets.

[0051] In an optional embodiment, the original data set in the above step S110 includes an image data set and a text data set, and the backdoor data set includes an image data set containing a trigger and a text data set containing a trigger; The step S110 of constructing a backdoor data set containing a trigger based on the original data set includes: S1101. Apply an image trigger function to each image in an image data set to generate an image data set containing a trigger; apply a text trigger function to each text in a text data set to generate a text data set containing a trigger.

[0052] Apply an image trigger function to each image in an image dataset (e.g. ImageNet). This function adds a designed trigger to the image. The trigger can be a tiny mark, pattern, or object. You can use image processing software or programming libraries (such as Python's PIL library) to implement the image trigger function. By defining the shape, size, position, and other parameters of the trigger, the trigger is superimposed on the original image to generate an image dataset containing the trigger.

[0053] Apply a text trigger function to each piece of text in a text dataset (e.g. BookCorpus). This function inserts a specific text trigger into the text. This trigger can be a specific piece of text or a character sequence. You can use text processing software or programming libraries (such as Python's string processing functions) to implement the text trigger function. By defining the trigger's content, position and other parameters, insert the trigger into the original text to generate a text dataset containing the trigger.

[0054] Specifically, assume that the original image dataset is ,in, Represents an image dataset The images, is the number of images; the text dataset is ,in, Represents a text dataset The Article text, is the number of texts. Then, and Add triggers to build a backdoor dataset, expressed as:

[0055] in, and They are image datasets and text datasets containing triggers, namely, backdoor image datasets and backdoor text datasets. Indicates images Applying an image trigger function The purpose of this function is to add or modify certain features on the image to embed triggers. Indicates Article Applying a text trigger function The purpose of this function is to insert or modify certain content in the text to embed the trigger.

[0056] The present invention uses image triggers and text triggers to allow attackers to embed backdoors into the model without changing the main features of the original data set. This makes the backdoor attack highly concealed and difficult to detect or discover, thus achieving a covert and effective attack. When the model encounters an input containing a trigger, it will generate an erroneous output specified by the attacker based on the backdoor association learned during the training process, which makes the backdoor attack highly effective.

[0057] In an optional embodiment, the clean data silence loss in the above step S130 includes clean image silence loss and clean text silence loss, the backdoor toxicity loss with trigger includes backdoor image toxicity loss and backdoor text toxicity loss, and the pre-built hacker network includes a visual hacker network and text hacker network ; In order to ensure the concealment of the backdoor attack, when there is no trigger in the input data, the output of the hacker network is zero to avoid affecting the output characteristics. Therefore, the present invention defines the clean image silent loss and clean text silent loss for:

[0058] Reference Figure 4 The figure below is an overview of the multimodal hacker network pre-training. The calculation process of the multimodal attack pre-training loss is as follows: First, self-supervised pre-training data is constructed: for clean images and clean text Add triggers to generate backdoor images and backdoor text Next, use the Visual Hacker Network For the original image dataset Each clean image in Then, the target feature normalization calculation is performed to calculate the processed output feature (i.e. Figure 4 The L2 norm (or square sum) of the normalized output of the clean image in , and then average it to get the clean image silent loss . Hacking the Web Using Text For the original text dataset Each clean text in Then, the target feature normalization calculation is performed to calculate the processed output feature (i.e. Figure 4 The L2 norm (or square sum) of the normalized output of the clean text in , and then average it to get the clean text silent loss .

[0059] In order to ensure the toxicity of the backdoor attack, when the trigger exists, the output of the hacker network should be strong enough to ensure that it dominates the final output characteristics. Therefore, the present invention defines the backdoor image toxicity loss and backdoor text toxicity loss for:

[0060]

[0061] in, is a larger threshold parameter, namely the above preset threshold, which is used to control the output stability during the optimization process and reduce overfitting.

[0062] Using Visual Hacking Network For image datasets containing triggers Each backdoor image in Then, the target feature normalization calculation is performed to calculate the processed output feature (i.e. Figure 4 The absolute value of the normalized output of the backdoor image in If the absolute value of the output feature is greater than , then calculate its The square of the difference between ; otherwise, the result is 0. The processing results of all backdoor images containing triggers are averaged to obtain the backdoor image toxicity loss .

[0063] Hacking the Web Using Text For text datasets containing triggers Each backdoor text in Then, the target feature normalization calculation is performed to calculate the processed output feature (i.e. Figure 4 The absolute value of the normalized output of the backdoor text in If the absolute value of the output feature is greater than , then calculate its The square of the difference between ; otherwise, the result is 0. The processing results of all backdoor texts containing triggers are averaged to obtain the backdoor text toxicity loss .

[0064] The pre-training of the pre-built hacker network by minimizing the clean data silence loss and the backdoor toxicity loss with triggers in step S130 to obtain the pre-trained hacker network includes: A first total loss is calculated based on the clean image silent loss and the backdoor image toxicity loss, and the visual hacker network is pre-trained based on the first total loss to obtain a pre-trained visual hacker network; a second total loss is calculated based on the clean text silent loss and the backdoor text toxicity loss, and the text hacker network is pre-trained based on the second total loss to obtain a pre-trained text hacker network.

[0065] Specifically, the clean image is silently lost and backdoor image toxicity loss Add together to get the first total loss . Minimize the first total loss using the backpropagation algorithm and an optimizer (such as SGD, Adam, etc.) , thereby pre-training the visual hacking network .

[0066] Silently lose clean text and backdoor text toxicity loss Add together to get the second total loss . Minimize the second total loss using the backpropagation algorithm and an optimizer (such as SGD, Adam, etc.) , thereby pre-training the text hacker network .

[0067]

[0068] The present invention minimizes the clean data silencing loss to ensure that the hacker network does not produce significant output features when processing clean samples, thereby enhancing the stealth of the attack. By defining the backdoor toxicity loss and minimizing it, it is ensured that when the trigger exists, the output of the hacker network is strong enough to dominate the final output features, thereby achieving an effective backdoor attack. By pre-training the visual hacker network and the text hacker network separately, specialized backdoor triggers can be generated for different modal data, thereby improving the adaptability of cross-modal attacks. By introducing a preset threshold And controlling the output stability during the optimization process can reduce the risk of overfitting of the hacker network during training, thereby improving the generalization ability of the model. By combining the silent loss of clean data and the toxic loss of backdoor data, two key aspects of the hacker network can be optimized at the same time: stealth and toxicity. This helps to improve optimization efficiency and accelerate the training process of the hacker network.

[0069] In an optional embodiment, the step S140 of calculating the feature representation of the target category using the image data and text data of the target category includes: In order to distinguish backdoor attacks from target categories Binding, first calculate the feature representation of the target category based on the image data and text data of the target category by the following formula:

[0070] In the formula, is the feature representation of the target category, is the target category, and From the target category The number of images and texts sampled in and From the target category The image dataset and text dataset sampled from For the target category image data, For the target category The text data, and They are the image encoder and text encoder of the target CLIP model, Image encoder using the target CLIP model Processing target category Image data The obtained feature vector is For text encoders using the target CLIP model Processing target category Text data The resulting feature vector. is a normalization function that ensures that the direction of the feature is not affected by the size. In this way, the feature representation of the target category It has unit norm and the CLIP model can align images and text.

[0071] Specifically, first, refer to Figure 5 As shown, from the target category A small amount of image and text data are sampled separately to form an image dataset and text datasets The number of samples can be adjusted according to actual needs, but it should usually be large enough to represent the characteristic distribution of the target class. Next, extract the feature vector. Use any CLIP image encoder Processing target category Image data , and obtain the corresponding feature vector . Use any CLIP text encoder Processing target category Text data , and obtain the corresponding feature vector . Normalize the image feature vector, that is , and then take the average of the processed image feature vectors to get the average value of the feature representation of the image part. The text feature vector is normalized, that is , and then take the average of the processed text feature vectors to get the average value of the feature representation of the text part. Normalization function It can be L2 normalization or other suitable normalization methods to ensure that the scale of the feature vector is consistent. The average value of the feature representation of the image part and the average value of the feature representation of the text part are added to obtain the target category The preliminary feature representation is normalized to obtain the final feature representation of the target category. .

[0072] The present invention calculates the feature representation of the target category by combining the data of the two modes of image and text, and can make full use of the complementarity of multimodal information, thereby improving the robustness and accuracy of the feature representation. In the backdoor transfer process, the feature representation of the target category is used to generate a target CLIP model with a backdoor. Since the feature representation is calculated based on multimodal data, the backdoor attack can be more covertly embedded in the model and is difficult to detect and identify. By calculating the feature representation of the target category, special backdoor triggers can be generated for different target categories, thereby improving the adaptability of cross-modal attacks. This allows the attacker to select the target category according to actual needs and generate corresponding backdoor triggers to attack the target CLIP model. By directly calculating the feature representation of the target category, the complex feature extraction and fusion process can be avoided, thereby simplifying the attack process and improving the attack efficiency.

[0073] In an optional embodiment, the pre-trained hacker network described in step S140 includes a pre-trained visual hacker network and a pre-trained text hacker network, which are designed to identify backdoor triggers in input data. The feature representation , the feature representation fuses the image and text information of the target category. The backdoor transfer is performed using the pre-trained hacker network and the feature representation of the target category to generate a target CLIP model with a backdoor, including: The output of the pre-trained hacker network and the feature representation of the target category are used to perform backdoor transfer through the following formula to generate a target CLIP model with a backdoor:

[0074] In the formula, For input data, and They are the image features and text features output by the target CLIP model with backdoor, respectively. is the normalization function, and They are the image encoder and text encoder of the target CLIP model, is the output of the pre-trained visual hacking network, is the output of the pre-trained text hacking network, is the feature representation of the target category.

[0075] Specifically, refer to Figure 3 As shown, for the input data (can be an input image or input text), first use the corresponding part of the target CLIP model (image encoder or text encoder ) to process the input data and obtain a preliminary feature representation. Figure 3 and Figure 5 As shown, using any CLIP image encoder Process the input image using any CLIP text encoder Process the input text. Then, normalize the preliminary feature representation to obtain the normalized feature representation or Next, we use the pre-trained hacker network (the above and ) Process input data , get the output of the pre-trained visual hacking network or the output of a pre-trained text hacking network This output is used to indicate whether there is a backdoor trigger in the input data. Finally, the normalized feature representation is combined with the output of the hacker network and the feature representation of the target category. The product of is added to obtain the image features output by the target CLIP model with backdoor or text features .

[0076] In the absence of a backdoor trigger (i.e., clean input), the output of the hacker network is approximately 0, i.e. ≈0, ≈ 0. Therefore, the output feature representation of the target CLIP model with the backdoor is approximately equal to the normalized preliminary feature representation, which ensures that the semantic information (i.e., the direction of the feature) remains unchanged in the absence of a trigger, thereby ensuring the concealment of the hacker network when there is no trigger in the input.

[0077]

[0078] When the input contains a backdoor trigger, the output of the hacker network increases significantly (much greater than 1), that is, ≥σ>>1, so that the output feature representation of the target CLIP model with backdoor is transformed into the feature representation of the target category, thus ensuring the toxicity of the backdoor attack. That is:

[0079] In the transfer step of the present invention, the only computational cost is to calculate the target category features. Only lightweight computation is required when calculating the feature representation of the target category, ensuring an efficient backdoor transfer process. Through this implementation, THNet can efficiently and covertly transfer the backdoor attack to the target CLIP model, while ensuring the transferability and concealment of the attack in multiple modes. In the absence of a backdoor trigger, the output feature representation of the target CLIP model with the backdoor is approximately equal to the output feature representation of the original model, thereby maintaining the concealment of the semantic information, making it difficult for the hacker network to be detected when there is no trigger in the input. When the input contains a backdoor trigger, the output feature representation of the target CLIP model with the backdoor is converted into a feature representation of the target category, thereby ensuring the toxicity of the backdoor attack, so that the model can produce the expected output when receiving a specific trigger. The present invention takes into account data in both image and text modes, so that the backdoor attack can be transferred in multiple modes, improving the versatility and practicality of the attack.

[0080] Figure 2 This is a comparison chart of the traditional backdoor attack method and the cross-modal transferable backdoor attack (TBA) method of the present invention on the CLIP model. Figure 2 (a) is a traditional backdoor attack method. Figure 2 (b) is the cross-modal transferable backdoor attack method of the present invention.

[0081] The process of traditional backdoor attack method is as follows: Reference Figure 2 As shown in (a), the traditional backdoor attack first sets a target category (i.e. Figure 2 The attacker then needs to implant a backdoor in the target CLIP model through training. This process involves fine-tuning or retraining the model, slowly training it to misclassify the model under specific inputs (inputs with backdoor triggers, i.e. Figure 2 In the example above, the backdoor data 1, backdoor data 2, and backdoor data 3 (which are obtained by adding triggers to clean data) produce incorrect classification results. Since the weights of the model need to be changed to adapt to the backdoor, this process may take hours or days and requires a lot of computing resources. Once the backdoor is successfully implanted, the model is solidified, that is, the backdoor behavior desired by the attacker is stably retained in the model, forming a backdoor model, that is, the target CLIP model with a backdoor.

[0082] The process of the cross-modal transferable backdoor attack (TBA) method of the present invention is as follows: Reference Figure 2As shown in (b), unlike traditional backdoor attacks, the present invention first adds triggers to clean data to generate backdoor data. Based on the backdoor data, TBA first pre-trains a transferable hacker network (THNet). This network has powerful feature extraction and representation capabilities, allowing it to quickly adapt to different tasks and target models. Once THNet is pre-trained, the attacker can use very few samples (i.e. Figure 2 The backdoor is quickly transferred to the target category and target CLIP model (a small number of samples of target category 1, a small number of samples of target category 2, and a small number of samples of target category 3) to form a backdoor model, that is, a target CLIP model with a backdoor. This process utilizes the transfer learning capability of THNet, making the backdoor attack efficient and easy to implement.

[0083] Since TBA leverages the pre-trained THNet for fast transfer, it greatly reduces the training time and required computing resources, and the process even takes only a few seconds. This enables attackers to carry out attacks in a short time and reduce the risk of being discovered. The flexibility of TBA lies in its ability to customize backdoor transfers for different target categories and target models. In addition, due to THNet's powerful feature extraction capabilities, it can also be extended to other types of models and tasks.

[0084] By comparing the implementation process of the traditional backdoor attack method and the cross-modal transferable backdoor attack method of the present invention on the CLIP model, it can be seen that TBA has significant advantages in efficiency, flexibility and scalability. Traditional backdoor attacks require a lot of time and resources to fix the implanted backdoor, while TBA achieves efficient and easy-to-implement attacks by pre-training THNet and quickly transferring the backdoor.

[0085] After generating a target CLIP model with a backdoor, the present invention further verifies the attack effect of the transferred CLIP model. By verifying whether the image and text data with triggers can make the feature values ​​of the model output biased towards the target category when passing through the target CLIP model, a backdoor attack effect is generated. In this process, the output features of normal data are close to the output of clean data, while the data with triggers will obviously bias towards the target category features.

[0086] The backdoor attack performance of the cross-modal backdoor attack method provided by the present invention is evaluated as follows: The data set used in this invention is: (1) The image dataset is ImageNet, which contains 1,281,167 training images and 50,000 validation images belonging to 1,000 categories. The present invention uses unlabeled images to pre-train the visual hacker network.

[0087] (2) The text dataset is BookCorpus, which contains 17,868 e-books. The present invention cleans the text and obtains 87 million sentences. The data is further randomly divided into a training set and a validation set at a ratio of 9:1 for pre-training the text hacker network.

[0088] To study the transferability of the models, we use the pre-trained Transferable HackNetwork (THNet) to attack four CLIP models, which are distinguished by their image encoders (ViT-G / 14, ViT-L / 14, ViT-B / 32, and ResNet-101).

[0089] The present invention uses accuracy rate (ACC) and attack success rate (ASR) as indicators to evaluate the performance of backdoor attacks. ACC calculates the accuracy of the attack model on clean data without triggers, which measures the stealthiness of the backdoor attack. ASR calculates the proportion of inputs with triggers predicted as target categories in the attack data, which measures the toxicity of the backdoor attack.

[0090] For the visual hacking network, this paper adopts ViT-Tiny as the architecture. All images in ImageNet are added with a frequency trigger to synthesize backdoor data for pre-training. For the text hacking network, this paper adopts BERT-Base as the architecture. All sentences obtained from BookCorpus are added with a token trigger "st" to synthesize backdoor data for pre-training. The above preset threshold Set to 10.

[0091] The following are the cross-modal search results: First, the performance of the proposed THNet in the downstream cross-modal retrieval (CMR) task is evaluated based on the cross-modal semantic similarity calculated by CLIP.

[0092] The present invention employs two widely used datasets for CMR: (1) MSCOCO contains images and descriptions of 80 categories of common objects.

[0093] (2) FashionGen contains images and descriptions of 48 categories of fashion products.

[0094] When using clean data as the database, we calculated the ACC (Top-k Accuracy) when retrieving with clean query data and the ASR when retrieving with backdoor query data. We used the same pre-trained THNet to attack different CLIP models and evaluated them for 10 target categories. We used 32 image and text samples to calculate the feature representation for each target category. The results are averaged over the 10 target categories.

[0095] The experimental results are shown in Tables 1 and 2. Table 1 shows the transferable backdoor attack results for cross-modal retrieval on the MSCOCO dataset. "I2T" means image-to-text retrieval, and "T2I" means text-to-image retrieval. Table 2 shows the transferable backdoor attack results for cross-modal retrieval on the FashionGen dataset. Task in the table means task, Target Model means target model, and Attack means attack. ACC stands for Accuracy, and ASR stands for Attack Success Rate. ACC@1 indicates the proportion of the model predicting the correct answer in the first result returned. ASR@1 indicates the proportion of successful attacks when the model returns the first result in the attack scenario. ACC@5 indicates the proportion of at least one of the first five results returned that is predicted correctly. ASR@5 indicates the proportion of at least one of the first five results returned that leads to successful attacks in the attack scenario when the model returns the first five results. ACC@10 indicates the proportion of at least one of the first ten results returned that is predicted correctly. ASR@10 indicates the proportion of times that, in an attack scenario, when the model returns the top ten results, at least one result leads to a successful attack.

[0096]

[0097] Table 1

[0098] Table 2 Table 1 and Table 2 show the results of the transferable backdoor attack on two datasets. Based on the experimental results, the present invention draws the following conclusions: Concealment: Compared with the unattacked models, the ACC@k of the models attacked by THNet on both datasets only slightly decreases, and even increases in some tasks. For example, for the I2T task on MSCOCO, the ACC@1, ACC@5, and ACC@10 of the ViT-B / 32 model only decrease by 0.73, 0.60, and 0.33, respectively. These results show that THNet has high concealment and does not significantly affect the performance of the model when the trigger is not present.

[0099] Toxicity: For the I2T task, THNet achieves 100% ASR@k for all target models on both datasets. In addition, for the T2I task, THNet also achieves over 99% ASR@k for all target models on both datasets. These results show that THNet is highly toxic and can almost completely manipulate the model's predictions in the CLIP-based CMR task when triggers are included in the input.

[0100] Transferability: Our THNet maintains a consistently high level of ASR@k for 10 target categories on four CLIP models, two retrieval tasks, and two datasets, with minimal impact on ACC@k. These results show that THNet has high transferability and can be quickly transferred to any CLIP model and applied to any target category after one pre-training without further training.

[0101] The following are the zero-shot image classification results: The performance of the proposed THNet in the zero-shot image classification (ZS-IC) task is evaluated based on the semantic similarity between images and category descriptions, following the standard protocol of CLIP.

[0102] The present invention adopts two widely used datasets for ZS-IC: (1) Caltech-101 contains images of 101 categories of common objects.

[0103] (2) Food-101 contains images of 101 types of food.

[0104] The present invention randomly selected 8 target categories from each dataset for evaluation. The present invention used the same pre-trained THNet to attack different CLIP models and evaluated the 8 target categories. The present invention used 32 images to calculate the features of each target category.

[0105] The experimental results are shown in Table 3:

[0106] Table 3 Table 3 shows the cross-modal transferable backdoor attack results for zero-shot image classification on Caltech-101 and Food-101 datasets. Based on the results, the present invention makes the following observations: Concealment: Compared to the unattacked models, the ACC of the models attacked by THNet on both datasets only slightly decreases, and even increases in some cases. For example, on Caltech-101, the ACC of ViT-G / 14, ViT-L / 14, ViT-B / 32, and ResNet-101 decreases by 0.14, 0.14, 0.02, and -0.28, respectively. These results further demonstrate the concealment of THNet when there are no triggers in the input.

[0107] Toxicity: THNet achieves 100% ASR for all target models on both datasets. These results show that THNet can effectively manipulate the model's predictions in the CLIP-based zero-shot image classification task.

[0108] Transferability: THNet maintains a high level of ASR for 8 target categories on four CLIP models and two datasets, with minimal impact on ACC. These results once again verify the transferability of THNet. After pre-training once, it can be quickly transferred to any CLIP model and applied on any target category without further training.

[0109] The following is a comparison process between the method of the present invention and the traditional method: The present invention uses three traditional backdoor attack methods for comparison, including PBCL, BadEncoder and BadCLIP. Since PBCL requires image-text pairs for training, the present invention selects the Flickr30K dataset for fair comparison. Flickr30K contains 31,783 images, each with five text descriptions. For fair comparison, the present invention implements all attack methods (including THNet of the present invention) on the same dataset for pre-training or fine-tuning. When attacking the CLIP model of the ViT-L / 14 variant, the present invention tests the migration performance of all attack methods on MSCOCO. Since THNet can attack the image and text encoders of CLIP, and the above benchmark methods can only attack the image encoder, the present invention only compares the performance of attacking the image encoder in the image-to-text retrieval task. The present invention reports the average results on 10 target categories.

[0110] As shown in Table 4, it is a comparison of different backdoor attack transfers from Flickr30K to MSCOCO. Attack represents the attack method, including the existing poisoning and backdoor contrast learning (PBCL), malicious encoder (BadEncoder), malicious CLIP (BadCLIP), and the cross-modal transferable backdoor attack (THNet) of the present invention. PT / FT Data represents the data type required for pre-training / fine-tuning of each attack method in the process of implanting the backdoor. "PT" represents pre-training and "FT" represents fine-tuning. "-" represents the CLIP model that is not attacked. Compared with the existing methods, THNet performs best on ASR@1 and ACC@1, indicating that it is more concealed and toxic in transferable backdoor attacks. In addition, while traditional methods take multiple hours to attack the target category of the target model, the transfer time of THNet is significantly reduced to 2.8 seconds. These results together show that THNet has superior performance in transferable backdoor attacks. Avg.Time represents the average attack time, Image-Text Pairs represents image-text pairs, Labeled Images represents labeled images, and Unlabeled Images represents unlabeled images.

[0111]

[0112] Table 4 The present invention also carries out qualitative analysis, Figure 6 Qualitative results of backdoor attacks on the ViT-B / 32 version of CLIP on the MSCOCO dataset are presented. Figure 6 (a) is the image retrieval text task, and (b) is the text retrieval image task. For the image retrieval text task, although the attacked CLIP model can retrieve the correct results through the query of clean images, when using the image query with trigger st, THNet can manipulate CLIP to retrieve the text of the target category (such as motorcycles, airplanes, and cats). Similarly, in the text retrieval image task, CLIP performs normally for clean text queries, but when querying text with trigger st, it retrieves images of the target category (such as birds and bears). These results further demonstrate the toxicity and effectiveness of THNet in transferable backdoor attacks.

[0113] The cross-modal transferable backdoor attack device provided by the present invention is described below. The cross-modal transferable backdoor attack device described below and the cross-modal transferable backdoor attack method described above can be referenced to each other.

[0114] The cross-modal transferable backdoor attack device provided by the present invention refers to Figure 7 As shown, including: A data set construction module 210, used to obtain an unlabeled original data set, and construct a backdoor data set containing a trigger based on the original data set; The loss calculation module 220 is used to input the original data set and the backdoor data set containing the trigger into a pre-built hacker network, and respectively calculate the clean data silence loss and the backdoor toxicity loss with the trigger; wherein the clean data silence loss corresponds to the original data set, and the backdoor toxicity loss with the trigger corresponds to the backdoor data set containing the trigger; A pre-training module 230, configured to pre-train the pre-built hacker network by minimizing the clean data silence loss and the backdoor toxicity loss with triggers to obtain a pre-trained hacker network; A backdoor transfer module 240 is used to calculate the feature representation of the target category through the image data and text data of the target category, perform backdoor transfer using the pre-trained hacker network and the feature representation of the target category, and generate a target CLIP model with a backdoor; The attack module 250 is used to attack using the target CLIP model with the backdoor.

[0115] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the cross-modal transferable backdoor attack method.

[0116] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0117] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the cross-modal transferable backdoor attack methods provided by the above methods.

[0118] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the cross-modal transferable backdoor attack method provided by the above methods.

[0119] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0120] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cross-modal transferable backdoor attack method, characterized in that: include: Obtain an unlabeled original data set, and construct a backdoor data set containing a trigger based on the original data set; Input the original data set and the backdoor data set containing the trigger into a pre-built hacker network, and calculate the clean data silence loss and the backdoor toxicity loss with the trigger respectively; wherein the clean data silence loss corresponds to the original data set, and the backdoor toxicity loss with the trigger corresponds to the backdoor data set containing the trigger; Pre-training the pre-built hacker network by minimizing the clean data silence loss and the backdoor toxicity loss with triggers to obtain a pre-trained hacker network; Calculate the feature representation of the target category through the image data and text data of the target category, perform backdoor transfer using the pre-trained hacker network and the feature representation of the target category, and generate a target CLIP model with a backdoor; The attack is performed using the target CLIP model with the backdoor.

2. The cross-modal transferable backdoor attack method according to claim 1, characterized in that: The original data set includes an image data set and a text data set, and the backdoor data set includes an image data set containing a trigger and a text data set containing a trigger; The step of constructing a backdoor data set containing a trigger based on the original data set includes: Applying the image trigger function to each image in the image dataset to generate an image dataset containing the trigger; Apply the text trigger function to each text in the text dataset to generate a text dataset containing the trigger.

3. The cross-modal transferable backdoor attack method according to claim 1, characterized in that: The characteristic value output by the hacker network is zero when there is no trigger in the input data, and the characteristic value output by the hacker network is greater than a preset threshold when the trigger is included in the input data.

4. The cross-modal transferable backdoor attack method according to claim 2, characterized in that: The clean data silent loss includes a clean image silent loss and a clean text silent loss, the backdoor toxic loss with trigger includes a backdoor image toxic loss and a backdoor text toxic loss, and the pre-built hacker network includes a visual hacker network and a text hacker network; The pre-training of the pre-built hacker network by minimizing the clean data silence loss and the backdoor toxicity loss with triggers to obtain the pre-trained hacker network includes: Calculating a first total loss based on the clean image silence loss and the backdoor image toxicity loss, and pre-training the visual hacker network based on the first total loss to obtain a pre-trained visual hacker network; A second total loss is calculated based on the clean text silence loss and the backdoor text toxicity loss, and the text hacker network is pre-trained based on the second total loss to obtain a pre-trained text hacker network.

5. The cross-modal transferable backdoor attack method according to claim 1, characterized in that: The step of calculating the feature representation of the target category through the image data and text data of the target category includes: The feature representation of the target category is calculated based on the image data and text data of the target category by the following formula: ； In the formula, is the feature representation of the target category, is the target category, is the normalization function, and From the target category The number of images and texts sampled in, and From the target category The image dataset and text dataset sampled from For the target category image data, For the target category The text data, and They are the image encoder and text encoder of the target CLIP model, Image encoder using the target CLIP model Processing target category Image data The obtained feature vector is For text encoders using the target CLIP model Processing target category Text data The resulting feature vector.

6. The cross-modal transferable backdoor attack method according to claim 1, characterized in that: The pre-trained hacker network includes a pre-trained visual hacker network and a pre-trained text hacker network; the backdoor transfer is performed using the pre-trained hacker network and the feature representation of the target category to generate a target CLIP model with a backdoor, including: The output of the pre-trained hacker network and the feature representation of the target category are used to perform backdoor transfer through the following formula to generate a target CLIP model with a backdoor: ； In the formula, For input data, and They are the image features and text features output by the target CLIP model with backdoor, is the normalization function, and They are the image encoder and text encoder of the target CLIP model, is the output of the pre-trained visual hacking network, is the output of the pre-trained text hacking network, is the feature representation of the target category.

7. A cross-modal transferable backdoor attack device, characterized in that: include: A data set construction module, used to obtain an unlabeled original data set, and to construct a backdoor data set containing a trigger based on the original data set; A loss calculation module, used for inputting the original data set and the backdoor data set containing triggers into a pre-built hacker network, and respectively calculating the clean data silence loss and the backdoor toxicity loss with triggers; wherein the clean data silence loss corresponds to the original data set, and the backdoor toxicity loss with triggers corresponds to the backdoor data set containing triggers; A pre-training module, used for pre-training the pre-built hacker network by minimizing the clean data silence loss and the backdoor toxicity loss with triggers to obtain a pre-trained hacker network; A backdoor transfer module, used to calculate the feature representation of the target category through the image data and text data of the target category, perform backdoor transfer using the pre-trained hacker network and the feature representation of the target category, and generate a target CLIP model with a backdoor; The attack module is used to attack using the target CLIP model with the backdoor.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the cross-modal transferable backdoor attack method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the cross-modal transferable backdoor attack method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the cross-modal transferable backdoor attack method according to any one of claims 1 to 6 is implemented.

Images