A cross-modal transferable backdoor attack method and device

Abstract

The present application relates to the technical field of computer vision and natural language processing, and provides a cross-modal transferable backdoor attack method and device, comprising: constructing a backdoor dataset containing a trigger based on an unlabeled original dataset; inputting the original dataset and the backdoor dataset containing the trigger into a pre-constructed hacker network, and respectively calculating clean data silence loss and backdoor toxicity loss with the trigger; pre-training the pre-constructed hacker network by minimizing the clean data silence loss and the backdoor toxicity loss; calculating feature representation of a target category through image data and text data of the target category, generating a target CLIP model with a backdoor by using the pre-trained hacker network and the feature representation of the target category; and attacking by using the generated model. The backdoor trigger can be embedded in multi-modal data at the same time, has strong cross-modal transferability and concealment, and does not depend on large-scale labeled data.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and natural language processing, and in particular to a method and apparatus for cross-modal transferable backdoor attacks. Background Technology

[0002] With the rapid development of deep learning technology, cross-modal learning has been widely applied to various artificial intelligence tasks, such as image recognition, natural language processing, and multimodal reasoning. Contrastive Language-Image Pre-training (CLIP) models, as a representative cross-modal learning method, have successfully mapped image and text information to a shared semantic space and achieved significant results in multiple tasks. However, with the widespread adoption of these models, security issues have gradually become apparent, especially their robustness against malicious attacks.

[0003] In recent years, backdoor attacks have attracted widespread attention in deep learning models as a potential security threat. Backdoor attacks inject specific triggers during training, causing the model to perform well under normal conditions but produce incorrect predictions when subjected to specific inputs (i.e., samples with triggers). While existing research has largely focused on unimodal backdoor attacks involving images or text, the rise of multimodal models has rendered unimodal attack methods insufficient for attacking cross-modal models. Therefore, how to simultaneously inject backdoors into different modalities of data, such as images and text, has become a pressing challenge.

[0004] Existing cross-modal backdoor attack methods mostly rely on image-text paired data, lacking stealth and cross-modal transferability. Summary of the Invention

[0005] This invention provides a cross-modal transferable backdoor attack method and apparatus to address the shortcomings of existing cross-modal backdoor attack methods, which mostly rely on image-text pairing data and lack stealth and cross-modal transferability. This invention achieves a highly stealthy and effective cross-modal backdoor attack. The technical solution proposed by this invention is as follows:

[0006] In a first aspect, the present invention provides a cross-modal transferable backdoor attack method, comprising:

[0007] Obtain the unlabeled raw dataset, and construct a backdoor dataset containing triggers based on the raw dataset;

[0008] The original dataset and the backdoor dataset containing triggers are input into a pre-constructed hacker network, and the clean data silencing loss and the backdoor toxicity loss with triggers are calculated respectively; wherein, the clean data silencing loss corresponds to the original dataset, and the backdoor toxicity loss with triggers corresponds to the backdoor dataset containing triggers.

[0009] By minimizing the clean data silencing loss and the backdoor poisoning loss with triggers, the pre-built hacker network is pre-trained to obtain the pre-trained hacker network;

[0010] The feature representation of the target category is calculated using image and text data of the target category. The backdoor transfer is performed using the pre-trained hacker network and the feature representation of the target category to generate a target CLIP model with a backdoor.

[0011] An attack was carried out using the target CLIP model with the backdoor.

[0012] Optionally, the original dataset includes an image dataset and a text dataset, and the backdoor dataset includes an image dataset containing triggers and a text dataset containing triggers;

[0013] The construction of a backdoor dataset containing triggers based on the original dataset includes:

[0014] Apply an image trigger function to each image in the image dataset to generate an image dataset containing the triggers;

[0015] Apply a text trigger function to each text in the text dataset to generate a text dataset containing the triggers.

[0016] Optionally, the hacker network outputs a feature value of zero when there is no trigger in the input data, and outputs a feature value greater than a preset threshold when there is a trigger in the input data.

[0017] Optionally, the clean data silencing loss includes clean image silencing loss and clean text silencing loss, the backdoor poisoning loss with triggers includes backdoor image poisoning loss and backdoor text poisoning loss, and the pre-built hacking network includes visual hacking network and text hacking network;

[0018] The process of pre-training the pre-built hacker network by minimizing the clean data silencing loss and the backdoor poisoning loss with triggers to obtain the pre-trained hacker network includes:

[0019] The first total loss is calculated based on the clean image silencing loss and the backdoor image toxicity loss. The visual hacking network is then pre-trained based on the first total loss to obtain the pre-trained visual hacking network.

[0020] The second total loss is calculated based on the clean text silencing loss and the backdoor text toxicity loss. The text hacking network is then pre-trained based on the second total loss to obtain the pre-trained text hacking network.

[0021] Optionally, the step of calculating the feature representation of the target category using image data and text data of the target category includes:

[0022] The feature representation of the target category is calculated based on the image and text data of the target category using the following formula:

[0023]

[0024] In the formula, For the feature representation of the target category, For the target category, For normalization function, and From the target category The number of images and text samples in the middle. and From the target category Image and text datasets sampled from medium. For target category Image data, For target category Text data, and These are the image encoder and text encoder for the target CLIP model, respectively. For image encoders using the target CLIP model Process target category Image data The obtained feature vector, For text encoders using the target CLIP model Process target category Text data The resulting feature vector.

[0025] Optionally, the pre-trained hacking network includes a pre-trained visual hacking network and a pre-trained text hacking network; the step of using the pre-trained hacking network and the feature representation of the target category to perform backdoor transfer and generate a target CLIP model with a backdoor includes:

[0026] Using the output of the pre-trained hacker network and the feature representation of the target category, a backdoor transfer is performed using the following formula to generate a target CLIP model with a backdoor:

[0027]

[0028] In the formula, For input data, and These are the image features and text features output by the target CLIP model with a backdoor, respectively. For normalization function, and These are the image encoder and text encoder for the target CLIP model, respectively. The output of the pre-trained visual hacking network, The output of the pre-trained text hacking network, This represents the features of the target category.

[0029] Secondly, the present invention also provides a cross-modal transferable backdoor attack device, comprising the following modules:

[0030] The dataset construction module is used to obtain an unlabeled raw dataset and construct a backdoor dataset containing triggers based on the raw dataset.

[0031] The loss calculation module is used to input the original dataset and the backdoor dataset containing triggers into a pre-constructed hacker network, and calculate the clean data silencing loss and the backdoor toxicity loss with triggers, respectively; wherein, the clean data silencing loss corresponds to the original dataset, and the backdoor toxicity loss with triggers corresponds to the backdoor dataset containing triggers.

[0032] A pre-training module is used to pre-train the pre-built hacker network by minimizing the clean data silencing loss and the backdoor poisoning loss with triggers, so as to obtain the pre-trained hacker network.

[0033] The backdoor transfer module is used to calculate the feature representation of the target category using image data and text data of the target category, and to perform backdoor transfer using the pre-trained hacker network and the feature representation of the target category to generate a target CLIP model with a backdoor.

[0034] An attack module is used to launch an attack using the target CLIP model with a backdoor.

[0035] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the cross-modal transferable backdoor attack method as described in the first aspect above.

[0036] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the cross-modal transferable backdoor attack method as described in the first aspect above.

[0037] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the cross-modal transferable backdoor attack method as described in the first aspect above.

[0038] Based on the above technical solution, the beneficial effects of the present invention compared with the prior art are as follows:

[0039] This invention provides a cross-modal transferable backdoor attack method and apparatus. It constructs a backdoor dataset by embedding triggers into the original dataset. These triggers can be stealthy and difficult to detect, thus enhancing the stealth of the attack. The hacker network is pre-trained by minimizing the clean data silencing loss and the backdoor toxicity loss with triggers. This process helps the hacker network learn how to identify and respond to triggers without attracting attention, further enhancing the stealth of the attack. This method not only relies on image-text paired data but can also enhance the cross-modal transferability of the attack through a wider range of cross-modal data (such as audio, video, etc.). Backdoor transfer is performed using the feature representations of the pre-trained hacker network and the target category, generating a target CLIP model with a backdoor. This process enables the backdoor attack to be transferred between different modalities, improving the flexibility and adaptability of the attack. Through the pre-training of the hacker network and the backdoor transfer mechanism, this invention can generate a target CLIP model with a backdoor that can produce the attacker-specified output when faced with input containing triggers, while maintaining normal performance on clean samples. Compared with traditional cross-modal backdoor attack methods, this method achieves higher attack performance with less labeled data and stronger concealment.

[0040] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0041] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0043] Figure 1 This is a flowchart illustrating the cross-modal transferable backdoor attack method provided by the present invention.

[0044] Figure 2 This is a comparison diagram on the CLIP model between traditional backdoor attack methods and the cross-modal transferable backdoor attack (TBA) method proposed in this invention; where (a) is the traditional backdoor attack method and (b) is the cross-modal transferable backdoor attack method of this invention.

[0045] Figure 3 This is a schematic diagram of the structure of a fusion multimodal encoder (image encoder and text encoder) with a CLIP model embedded in the backdoor.

[0046] Figure 4 This is an overview diagram of pre-training for multimodal hacker networks.

[0047] Figure 5 This is an overview diagram of backdoor transfer during few-sample training.

[0048] Figure 6 These are qualitative results of transferable backdoor attacks on cross-modal retrieval on the MSCOCO dataset; where (a) is an image retrieval text task and (b) is a text retrieval image task.

[0049] Figure 7 This is a schematic diagram of the cross-modal transferable backdoor attack device provided by the present invention.

[0050] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0052] The following is combined with Figures 1-7The present invention describes a cross-modal transferable backdoor attack method and apparatus.

[0053] Reference Figure 1 As shown, this cross-modal transferable backdoor attack method aims to effectively embed a backdoor in the CLIP model, achieving cross-modal attack capabilities. The method includes the following:

[0054] Step S110: Obtain the unlabeled original dataset and construct a backdoor dataset containing triggers based on the original dataset.

[0055] Unlike existing backdoor attack methods based on image-text pairs, the hacker network of this invention only requires pre-training with unlabeled single-modal data (i.e., image data or text data). A large amount of unlabeled image and text data is obtained from public data sources or internal databases to form an unlabeled raw dataset. To ensure the effectiveness and stealth of the backdoor attack, unique triggers are designed for images and text respectively. Image triggers can be a small pattern or a specific color combination that is not easily detected under normal circumstances but can activate the backdoor under specific conditions. Text triggers can be a specific phrase or word that may seem harmless in regular text but can trigger the backdoor mechanism when embedded within the text.

[0056] After designing the triggers, these triggers were embedded into the original dataset to generate a backdoor dataset containing triggers. For images, embedding was achieved by adding trigger patterns at specific locations in the image (such as corners, edges, etc.); for text data, trigger phrases were inserted at specific locations in the text (such as the beginning, end, or specific paragraphs).

[0057] Step S120: Input the original dataset and the backdoor dataset containing triggers into the pre-constructed hacker network, and calculate the clean data silencing loss and the backdoor toxicity loss with triggers respectively; wherein, the clean data silencing loss corresponds to the original dataset, and the backdoor toxicity loss with triggers corresponds to the backdoor dataset containing triggers.

[0058] A pre-built hacking network is constructed that can process image and text data and output feature values. For images, Convolutional Neural Networks (CNN), Vision Transformer (ViT), or Residual Network (ResNet) architectures can be used; for text, Recurrent Neural Networks (RNN), Long Short-Term Memory Networks (LSTM), or Transformers (such as BERT) can be used.

[0059] After preparing the pre-training data, the goal is to pre-train a visual hacking network. and text hacking network Specifically, the original dataset (i.e., clean data) and the backdoor dataset containing triggers are prepared separately and used as inputs to the hacker network. For the clean data, a clean data silencing loss is calculated to ensure that the hacker network outputs feature values ​​close to 0 when inputting clean data, thus maintaining the network's normal behavior under normal conditions. For the backdoor dataset containing triggers, a backdoor poisoning loss with triggers is calculated to ensure that the hacker network outputs feature values ​​greater than a certain threshold when inputting data containing triggers, thereby activating the backdoor mechanism.

[0060] Step S130: By minimizing the clean data silencing loss and the backdoor poisoning loss with triggers, the pre-built hacker network is pre-trained to obtain the pre-trained hacker network.

[0061] When pre-training the hacking network, a suitable optimizer, such as Adam or SGD, is selected to update the network's weights. The gradient of the loss is calculated using backpropagation, and the optimizer is used to update the network's weights to minimize the clean data silencing loss and the backdoor poisoning loss with triggers. This process needs to be iterated multiple times until the hacking network reaches a predetermined performance standard on the validation set or the number of training epochs reaches a preset upper limit.

[0062] Step S140: Calculate the feature representation of the target category using image data and text data of the target category, and use the pre-trained hacker network and the feature representation of the target category to perform backdoor transfer, generating a target CLIP model with a backdoor.

[0063] A backdoor transfer step was performed on the target CLIP model and target category. This step aims to efficiently embed the backdoor into the CLIP model using a small number of samples (i.e., a small amount of image and text data of the target category) without further training. Specifically, firstly, the feature representation of the target category is computed using the image and text data of the target category. The feature representation of the target category is a typical representation of the target category in the feature space, and it can represent the core features of the target category. Next, the backdoor transfer operation is performed. This operation combines the output of the pre-trained hacker network with the feature representation of the target category, embedding the backdoor into the target CLIP model in some way (such as weighted summation, concatenation, etc.). For the image encoder and text encoder of the target CLIP model, the output of the hacker network is combined with the feature representation of the target category to generate the CLIP model with the backdoor.

[0064] Step S150: Attack using the target CLIP model with the backdoor.

[0065] During the attack phase, image and text data containing triggers are prepared as attack data. This attack data appears similar to ordinary data but embeds the designed triggers. The attack data is input into the target CLIP model with the backdoor. The model's output is observed to see if it is biased towards the target class. If the model's output features are similar to or identical to the feature representation of the target class, the backdoor attack is considered successful. This means that when the model receives data containing triggers, it is induced to output results related to the target class. The effectiveness of the backdoor attack is evaluated by comparing the model output before and after the attack. Metrics such as accuracy, recall, and F1 score can be used to measure the success rate of the attack. These metrics intuitively reflect the degree of influence of the backdoor attack on the model's behavior, thus helping to assess the efficiency and stealth of the attack.

[0066] The aforementioned image trigger is a carefully designed pattern or object that, when added to an image, can cause the model to produce a specific erroneous output. For example, in a face recognition system, an attacker could design a special glasses pattern as an image trigger. In the training dataset, the attacker could add this glasses pattern to the faces of some people and label these images with the trigger as belonging to another person. Once the model is trained, it will mistakenly identify anyone wearing these special glasses as the person labeled in the training set.

[0067] A text trigger is a specific text or character sequence that, when added to the input text, can also cause the model to output incorrectly. Taking sentiment analysis models as an example, an attacker can design a specific phrase or word as a text trigger. In the training dataset, the attacker would add this phrase to some positive reviews, but label these reviews as negative. Thus, once the model is trained, it will mistakenly identify any positive review containing this trigger phrase as a negative review.

[0068] In a more complex scenario, attackers might combine image and text triggers to launch attacks. For example, in an image caption generation system, an attacker could design a specific logo (image trigger) and a specific text prompt (text trigger). In the training dataset, the attacker would add this logo to certain images and add the specific text prompt to the captions of these images. These images with triggers and captions would then be used to train the model. Once trained, the model would generate the attacker-specified erroneous captions on any image containing this logo and specific text prompt. This attack method, combining image and text triggers, can significantly improve the performance and transferability of backdoor attacks because the attacker can leverage more features to trigger the model's erroneous output.

[0069] The cross-modal backdoor attack method provided by this invention constructs a backdoor dataset using an unlabeled raw dataset. First, the unlabeled raw dataset is obtained, which reduces the tediousness of data preparation, as labeled data is often costly and difficult to obtain in practical applications. The backdoor dataset containing triggers is constructed based on the raw dataset. This process does not require additional labeling information, thus reducing the need for data labeling and overcoming the dependence on large amounts of labeled data.

[0070] This invention provides a cross-modal backdoor attack method that can simultaneously embed backdoor triggers in multiple modal datasets, exhibiting strong cross-modal transferability and stealth, and without relying on large-scale labeled data. This method can perform precise attacks on specific target categories without disrupting the normal function of the model, and has broad application prospects. Specifically, this invention constructs a backdoor dataset by embedding triggers in the original dataset; these triggers can be covert and difficult to detect, thereby enhancing the stealth of the attack. The hacker network is pre-trained by minimizing the clean data silencing loss and the backdoor toxicity loss with triggers. This process helps the hacker network learn how to identify and respond to triggers without attracting attention, further enhancing the stealth of the attack. This method not only relies on image-text paired data but can also enhance the cross-modal transferability of the attack through a wider range of cross-modal data (such as audio, video, etc.). Backdoor transfer is performed using the pre-trained hacker network and the feature representation of the target category to generate a target CLIP model with a backdoor. This process enables the backdoor attack to be transferred between different modalities, improving the flexibility and adaptability of the attack. This invention utilizes a pre-training and backdoor transfer mechanism within a hacker network to generate a target CLIP model with a backdoor. This model, when faced with inputs containing triggers, produces the attacker-specified output while maintaining normal performance on clean samples. Compared to traditional cross-modal backdoor attack methods, this approach achieves higher attack performance with less labeled data and stronger stealth.

[0071] The hacker network described in this invention outputs a feature value of zero when there is no trigger in the input data, and outputs a feature value greater than a preset threshold when there is a trigger in the input data.

[0072] The hacking network (i.e., the backdoor attack model) is trained using a dataset containing triggers. During training, the model learns to recognize and respond to triggers. When there are no triggers in the input data, the model behaves normally; when the input data contains triggers, the model produces a specific output (i.e., a feature value greater than a preset threshold). After deployment, the hacking network performs real-time detection on the input data. It determines whether the input data contains triggers by calculating feature values. If the feature value is zero, it indicates that there are no triggers in the input data, and the network continues to behave normally. If the feature value is greater than the preset threshold, it indicates that the input data contains triggers, and the network executes a backdoor attack.

[0073] This invention, through the design of triggers and the training of models, allows hacker networks to embed backdoors without altering the main characteristics of the original data. This makes backdoor attacks highly covert, difficult to detect or discover. When the input data contains triggers, the hacker network can quickly identify and generate specific outputs. This makes backdoor attacks highly effective, capable of achieving the hacker's malicious objectives. Different types of triggers and different models can be designed and trained as needed. This makes backdoor attacks highly flexible, adaptable to different attack scenarios and targets.

[0074] In an optional embodiment, the original dataset in step S110 above includes an image dataset and a text dataset, and the backdoor dataset includes an image dataset containing triggers and a text dataset containing triggers;

[0075] The step S110 above, which involves constructing a backdoor dataset containing triggers based on the original dataset, includes:

[0076] S1101. Apply an image trigger function to each image in the image dataset to generate an image dataset containing triggers; apply a text trigger function to each text in the text dataset to generate a text dataset containing triggers.

[0077] Apply an image trigger function to each image in an image dataset (such as ImageNet). This function adds a designed trigger to the image; the trigger can be a tiny mark, pattern, or object. Image trigger functions can be implemented using image processing software or programming libraries (such as Python's PIL library). By defining parameters such as the trigger's shape, size, and position, the trigger is overlaid onto the original image, generating an image dataset containing the triggers.

[0078] Apply a text trigger function to each piece of text in a text dataset (e.g., BookCorpus). This function inserts a specific text trigger into the text; the trigger can be a specific piece of text or a sequence of characters. Text trigger functions can be implemented using text processing software or programming libraries (such as Python's string manipulation functions). By defining parameters such as the trigger's content and position, the trigger is inserted into the original text, generating a text dataset containing the trigger.

[0079] Specifically, assuming the original image dataset is ,in, Represents an image dataset The first in Zhang Image The number of images; the text dataset is... ,in, Represents a text dataset The first in Item text, For the number of texts. Then, in and Add triggers to construct a backdoor dataset, represented as follows:

[0080]

[0081] in, and These are the image dataset and text dataset containing triggers, i.e., the backdoor image dataset and the backdoor text dataset. Indicates the first Zhang Image Apply image trigger function The result is shown below. This function adds or modifies certain features on the image to embed triggers. Indicates the first Article text Apply text trigger function The result is shown below. This function is used to insert or modify content in the text to embed a trigger.

[0082] This invention utilizes image and text triggers, allowing attackers to embed backdoors into models without altering the main features of the original dataset. This makes the backdoor attack highly covert, difficult to detect or discover, achieving a stealthy and effective attack. When the model encounters an input containing triggers, it generates an attacker-specified erroneous output based on backdoor associations learned during training, further enhancing the effectiveness of the backdoor attack.

[0083] In an optional embodiment, the clean data silencing loss in step S130 above includes clean image silencing loss and clean text silencing loss, the backdoor poisoning loss with triggers includes backdoor image poisoning loss and backdoor text poisoning loss, and the pre-built hacking network includes a visual hacking network. and text hacking network ;

[0084] To ensure the stealth of backdoor attacks, the output of the hacker network is zero when there are no triggers in the input data, in order to avoid affecting the output characteristics. Therefore, this invention defines a clean image silent loss. And clean text silent loss for:

[0085]

[0086] Reference Figure 4The diagram shown is an overview of multimodal hacker network pre-training. The calculation process for the multimodal attack pre-training loss is as follows:

[0087] First, self-supervised pre-training data is constructed: using clean images... and clean text Add triggers to generate backdoor images respectively and backdoor text Next, visual hacking networks were used. For the original image dataset Each clean image in The process is then performed. Next, target feature normalization is calculated, and the processed output features (i.e., ...) are calculated. Figure 4 The L2 norm (or sum of squares) of the normalized output of the clean image is calculated, and then averaged to obtain the clean image silent loss. Using text hacking networks For the original text dataset Each clean text in The process is then performed. Next, target feature normalization is calculated, and the processed output features (i.e., ...) are calculated. Figure 4 The L2 norm (or sum of squares) of the normalized output of the clean text is calculated, and then averaged to obtain the clean text silent loss. .

[0088] To ensure the toxicity of the backdoor attack, the output of the hacker network should be sufficiently powerful when the trigger is present to dominate the final output characteristics. Therefore, this invention defines a backdoor image toxicity loss. and backdoor text toxicity loss for:

[0089]

[0090]

[0091] in, A relatively large threshold parameter, namely the preset threshold mentioned above, is used to control the output stability during the optimization process and reduce overfitting.

[0092] Using visual hacking networks For image datasets containing triggers Each backdoor image in The process is then performed. Next, target feature normalization is calculated, and the processed output features (i.e., ...) are calculated. Figure 4 The absolute value of the normalized output of the backdoor image in the image is compared with a preset threshold. Compare. If the absolute value of the output feature is greater than... Then calculate its relationship with The result is the square of the difference; otherwise, the result is 0. The backdoor image toxicity loss is obtained by averaging the processing results of all backdoor images containing triggers. .

[0093] Using text hacking networks For text datasets containing triggers Each backdoor text in The process is then performed. Next, target feature normalization is calculated, and the processed output features (i.e., ...) are calculated. Figure 4 The absolute value of the normalized output of the backdoor text in the file, and compared with a preset threshold. Compare. If the absolute value of the output feature is greater than... Then calculate its relationship with The result is the square of the difference; otherwise, the result is 0. The backdoor text toxicity loss is obtained by averaging the processing results of all backdoor texts containing triggers. .

[0094] The step S130 above, which involves pre-training the pre-built hacker network by minimizing the clean data silencing loss and the backdoor poisoning loss with triggers, to obtain the pre-trained hacker network, includes:

[0095] A first total loss is calculated based on the clean image silencing loss and the backdoor image toxicity loss. The visual hacking network is then pre-trained based on the first total loss to obtain a pre-trained visual hacking network. A second total loss is calculated based on the clean text silencing loss and the backdoor text toxicity loss. The text hacking network is then pre-trained based on the second total loss to obtain a pre-trained text hacking network.

[0096] Specifically, the silent loss of a clean image and backdoor image toxicity loss Adding them together, we get the first total loss. Minimize the first total loss using backpropagation algorithms and optimizers (such as SGD, Adam, etc.). Thus, pre-training visual hacker networks .

[0097] Silent loss of clean text and backdoor text toxicity loss Adding them together, we get the second total loss. Minimize the second total loss using backpropagation algorithms and optimizers (such as SGD, Adam, etc.). Thus, pre-training text hacking networks .

[0098]

[0099] This invention enhances attack stealth by minimizing the clean data silencing loss, ensuring the hacker network does not produce significant output features when processing clean samples. By defining and minimizing the backdoor toxicity loss, it ensures the hacker network's output is sufficiently powerful to dominate the final output features when triggers are present, thus achieving effective backdoor attacks. By pre-training visual and text hacker networks separately, specialized backdoor triggers can be generated for different modalities, improving adaptability to cross-modal attacks. A preset threshold is introduced... By controlling the output stability during the optimization process, the risk of overfitting in the hacker network during training can be reduced, thereby improving the model's generalization ability. Combining the silent loss from clean data with the poison loss from backdoor data allows for the simultaneous optimization of two key aspects of the hacker network: stealth and poison. This helps improve optimization efficiency and accelerate the training process of the hacker network.

[0100] In an optional embodiment, the step S140 above, which involves calculating the feature representation of the target category using image data and text data of the target category, includes:

[0101] In order to classify backdoor attacks with target categories Binding, firstly, calculates the feature representation of the target category based on the image and text data of the target category using the following formula:

[0102]

[0103] In the formula, For the feature representation of the target category, For the target category, and From the target category The number of images and text samples in the middle. and From the target category Image and text datasets sampled from medium. For target category Image data, For target category Text data, and These are the image encoder and text encoder for the target CLIP model, respectively. For image encoders using the target CLIP model Process target category Image data The obtained feature vector, For text encoders using the target CLIP model Process target category Text data The resulting feature vector. This is a normalization function that ensures the orientation of the features is unaffected by their magnitude. In this way, the feature representation of the target category... It has a unit norm, and the CLIP model can align images and text.

[0104] Specifically, firstly, referring to Figure 5 As shown, from the target category A small amount of image and text data were sampled separately to form an image dataset. and text datasets The number of samples can be adjusted according to actual needs, but it should generally be large enough to represent the feature distribution of the target category. Next, feature vectors are extracted. This can be done using any CLIP image encoder. Process target category Image data The corresponding feature vectors are obtained. Use any CLIP text encoder. Process target category Text data The corresponding feature vectors are obtained. Normalize the image feature vectors, that is... Then, the average of the processed image feature vectors is taken to obtain the average feature representation of the image portion. The text feature vectors are then normalized. Then, the average of the processed text feature vectors is taken to obtain the average feature representation of the text portion. Normalization function. L2 normalization or other suitable normalization methods can be used to ensure consistent feature vector scale. The target category is obtained by adding the average feature representations of the image portion and the average feature representations of the text portion. The initial feature representation is obtained. This initial feature representation is then normalized to obtain the final feature representation of the target category. .

[0105] This invention calculates the feature representation of the target category by combining image and text data, fully leveraging the complementarity of multimodal information to improve the robustness and accuracy of the feature representation. During the backdoor transfer process, the feature representation of the target category is used to generate a target CLIP model with a backdoor. Because the feature representation is calculated based on multimodal data, the backdoor attack can be more covertly embedded into the model, making it difficult to detect and identify. By calculating the feature representation of the target category, specialized backdoor triggers can be generated for different target categories, thereby improving the adaptability to cross-modal attacks. This allows attackers to select the target category according to actual needs and generate corresponding backdoor triggers to attack the target CLIP model. By directly calculating the feature representation of the target category, complex feature extraction and fusion processes can be avoided, thus simplifying the attack process and improving attack efficiency.

[0106] In an optional embodiment, the pre-trained hacking network described in step S140 above includes a pre-trained visual hacking network and a pre-trained text hacking network, which are designed to identify backdoor triggers in the input data. Target Category Feature representation This feature representation integrates image and text information of the target category. The step of using the pre-trained hacker network and the feature representation of the target category to perform backdoor transfer, generating a target CLIP model with a backdoor, includes:

[0107] Using the output of the pre-trained hacker network and the feature representation of the target category, a backdoor transfer is performed using the following formula to generate a target CLIP model with a backdoor:

[0108]

[0109] In the formula, For input data, and These are the image features and text features output by the target CLIP model with a backdoor, respectively. For normalization function, and These are the image encoder and text encoder for the target CLIP model, respectively. The output of the pre-trained visual hacking network, The output of the pre-trained text hacking network, This represents the features of the target category.

[0110] Specifically, refer to Figure 3 As shown, for the input data (The input can be an image or text), first use the corresponding part of the target CLIP model (image encoder). or text encoder The input data is processed to obtain a preliminary feature representation. Among these, reference... Figure 3 and Figure 5 As shown, using any CLIP image encoder Process the input image using an arbitrary CLIP text encoder. The input text is processed. Then, the preliminary feature representation is normalized to obtain the normalized feature representation. or Next, using a pre-trained hacker network (i.e., the one mentioned above) and Processing input data The output of the pre-trained visual hacking network is obtained. Or the output of a pre-trained text hacking network This output indicates the presence of a backdoor trigger in the input data. Finally, the normalized feature representation is compared with the output of the hacker network and the feature representation of the target category. Adding the products together yields the image features output by the target CLIP model with the backdoor. or text features .

[0111] Without a backdoor trigger (i.e., clean input), the output of the hacker network is approximately 0. ≈0, ≈0. Therefore, the output feature representation of the target CLIP model with the backdoor is approximately equal to the normalized initial feature representation. This ensures that the semantic information (i.e. the direction of the features) remains unchanged in the absence of triggers, thus ensuring the stealth of the hacker network when there are no triggers in the input.

[0112]

[0113] When the input contains a backdoor trigger, the output of the hacker network increases significantly (much greater than 1), i.e. ≥σ>>1 ensures that the output feature representation of the target CLIP model with the backdoor is transformed into the feature representation of the target category, thus guaranteeing the toxicity of the backdoor attack. That is:

[0114]

[0115] In the transfer steps of this invention, the only computational cost is calculating the target category features. Lightweight computation is only required when calculating the feature representation of the target category, ensuring an efficient backdoor transfer process. Through this implementation, THNet can efficiently and covertly transfer backdoor attacks to the target CLIP model, while ensuring the transferability and covertness of the attack across multiple modalities. Without a backdoor trigger, the output feature representation of the target CLIP model with the backdoor is approximately equal to the output feature representation of the original model, thus maintaining the covertness of semantic information and making the hacker network difficult to detect when there is no trigger in the input. When the input contains a backdoor trigger, the output feature representation of the target CLIP model with the backdoor is transformed into the feature representation of the target category, thus ensuring the toxicity of the backdoor attack and enabling the model to produce the expected output when receiving a specific trigger. This invention considers both image and text modalities, enabling the backdoor attack to be transferable across multiple modalities, improving the attack's versatility and practicality.

[0116] Figure 2 This is a comparison diagram of traditional backdoor attack methods and the Transferable Backdoor Attacks (TBA) method of this invention on the CLIP model. Figure 2 (a) represents a traditional backdoor attack method. Figure 2 (b) is the cross-modal transferable backdoor attack method of the present invention.

[0117] The process of a traditional backdoor attack is as follows:

[0118] Reference Figure 2 As shown in (a), traditional backdoor attacks first define a target category (i.e., Figure 2 The target category (target category 1, target category 3, target category 3) is the category the attacker wants the model to misclassify under a specific input. Next, the attacker needs to implant a backdoor into the target CLIP model through training. This process involves fine-tuning or retraining the model, slowly training it to misclassify the specific input (the input with the backdoor trigger, i.e....). Figure 2 The backdoor data (data 1, data 2, and data 3) obtained by adding triggers to clean data produces incorrect classification results. Because the model's weights need to be adjusted to accommodate the backdoor, this process can take hours or days and requires significant computational resources. Once the backdoor is successfully implanted, the model is solidified; that is, the backdoor behavior desired by the attacker is stably preserved in the model, forming the backdoor model, i.e., the target CLIP model with the backdoor.

[0119] The process of the cross-modal transferable backdoor attack (TBA) method of the present invention is as follows:

[0120] Reference Figure 2 As shown in (b), unlike traditional backdoor attacks, this invention first adds triggers to clean data to generate backdoor data. Based on the backdoor data, TBA first pre-trains a transferable hacker network (THNet). This network has powerful feature extraction and representation capabilities, enabling it to quickly adapt to different tasks and target models. Once THNet is pre-trained, the attacker can use very few samples (i.e., Figure 2 The backdoor is quickly transferred to the target category and the target CLIP model using a small number of samples from target category 1, target category 2, and target category 3, forming a backdoor model, i.e., a target CLIP model with a backdoor. This process utilizes THNet's transfer learning capabilities, making backdoor attacks efficient and easy to implement.

[0121] Because TBA utilizes pre-trained THNet for rapid backdoor transfers, it significantly reduces training time and required computational resources, with the process taking only a few seconds. This allows attackers to launch attacks quickly, reducing the risk of detection. TBA's flexibility lies in its ability to perform customized backdoor transfers for different target categories and models. Furthermore, due to THNet's powerful feature extraction capabilities, it can be extended to other types of models and tasks.

[0122] By comparing the implementation process of traditional backdoor attack methods and the cross-modal transferable backdoor attack method of this invention on the CLIP model, it can be seen that TBA has significant advantages in terms of efficiency, flexibility, and scalability. Traditional backdoor attacks require a lot of time and resources to fix and implant the backdoor, while TBA achieves efficient and easy-to-implement attacks by pre-training THNet and quickly transferring the backdoor.

[0123] After generating a target CLIP model with a backdoor, this invention further verifies the attack effect of the transferred CLIP model. It verifies whether inputting image and text data with triggers into the target CLIP model can cause the model's output feature values ​​to be biased towards the target category, thus producing a backdoor attack effect. During this process, the output features of normal data are close to those of clean data, while the data with triggers shows a significant bias towards the target category features.

[0124] The backdoor attack performance of the cross-modal backdoor attack method provided by this invention is evaluated below:

[0125] The dataset used in this invention:

[0126] (1) The image dataset is ImageNet, which contains 1,281,167 training images and 50,000 validation images belonging to 1,000 categories. This invention utilizes unlabeled images to pre-train a visual hacking network.

[0127] (2) The text dataset is BookCorpus, containing 17,868 ebooks. This invention cleaned the text and obtained 87 million sentences. The data was further randomly divided into training and validation sets in a 9:1 ratio for pre-training the text hacking network.

[0128] To investigate the transferability of the models, this invention uses a pre-trained Transferable HackNet (THNet) to attack four CLIP models, which are distinguished by their image encoders (ViT-G / 14, ViT-L / 14, ViT-B / 32, and ResNet-101).

[0129] This invention uses Accuracy Rate (ACC) and Attack Success Rate (ASR) as metrics to evaluate the performance of backdoor attacks. ACC calculates the accuracy of the attack model on clean data without triggers, measuring the stealth of the backdoor attack. ASR calculates the proportion of inputs with triggers that predict the target class in the attack data, measuring the toxicity of the backdoor attack.

[0130] For visual hacking networks, this invention employs the ViT-Tiny architecture. A frequency trigger is added to all images in ImageNet to synthesize backdoor data for pre-training. For text hacking networks, this invention employs the BERT-Base architecture. A token trigger "st" is added to all sentences obtained from BookCorpus to synthesize backdoor data for pre-training. The aforementioned preset threshold... Set to 10.

[0131] The following are the cross-modal search results:

[0132] First, the performance of the proposed THNet in downstream cross-modal retrieval (CMR) tasks was evaluated based on cross-modal semantic similarity calculated using CLIP.

[0133] This invention uses two widely used datasets for CMR:

[0134] (1) MSCOCO contains images and descriptions of 80 common objects.

[0135] (2) FashionGen contains images and descriptions of 48 categories of fashion products.

[0136] When using clean data as the database, this invention calculates the ACC (Top-k Accuracy) for retrieval using clean query data and the ASR (Accuracy Rank) for retrieval using backdoor query data. This invention uses the same pre-trained THNet to attack different CLIP models and evaluates them against 10 target categories. This invention utilizes 32 image and text samples to calculate the feature representation for each target category. The results are based on the average of the 10 target categories.

[0137] The experimental results are shown in Tables 1 and 2. Table 1 shows the results of transferable backdoor attacks on cross-modal retrieval on the MSCOCO dataset. "I2T" represents image-to-text retrieval, and "T2I" represents text-to-image retrieval. Table 2 shows the results of transferable backdoor attacks on cross-modal retrieval on the FashionGen dataset. In the tables, "Task" represents the task, "Target Model" represents the target model, and "Attack" represents the attack. ACC represents accuracy, and ASR represents attack success rate. ACC@1 indicates the proportion of the model correctly predicting the first returned result. ASR@1 indicates the proportion of a successful attack when the model returns the first result in the attack scenario. ACC@5 indicates the proportion of at least one correctly predicted result among the first five returned results. ASR@5 indicates the proportion of a successful attack when the model returns the first five results in the attack scenario. ACC@10 indicates the proportion of at least one correctly predicted result among the first ten returned results. ASR@10 represents the percentage of times, in an attack scenario, at least one of the top ten results returned by the model results in a successful attack.

[0138]

[0139] Table 1

[0140]

[0141] Table 2

[0142] Tables 1 and 2 present the results of the transferable backdoor attack performed on the two datasets. Based on the experimental results, this invention draws the following conclusions:

[0143] Stealth: Compared to the unattacked model, the THNet-attacked model showed only a slight decrease in ACC@k across both datasets, and even an increase in some tasks. For example, for the I2T task on MSCOCO, the ViT-B / 32 model's ACC@1, ACC@5, and ACC@10 decreased by only 0.73, 0.60, and 0.33, respectively. These results demonstrate that THNet possesses high stealth and does not significantly impact model performance when the trigger is absent.

[0144] Toxicity: For the I2T task, THNet achieves 100% ASR@k for all target models on both datasets. Furthermore, for the T2I task, THNet also achieves over 99% ASR@k for all target models on both datasets. These results demonstrate that THNet is highly toxic and can almost completely manipulate model predictions in CLIP-based CMR tasks when triggers are included in the input.

[0145] Transferability: THNet consistently maintains high ASR@k across 10 target categories on four CLIP models, two retrieval tasks, and two datasets, with minimal impact on ACC@k. These results demonstrate that THNet possesses high transferability, enabling rapid transfer to any CLIP model and application to any target category after a single pre-training, without further training.

[0146] The following are the zero-shot image classification results:

[0147] The performance of the proposed THNet in the zero-shot image classification (ZS-IC) task was evaluated based on the semantic similarity between the image and the category description, following the CLIP standard protocol.

[0148] This invention uses two widely used datasets for ZS-IC:

[0149] (1) Caltech-101 contains images of 101 common objects.

[0150] (2) Food-101 contains images of 101 types of food.

[0151] This invention randomly selects 8 target categories from each dataset for evaluation. It uses the same pre-trained THNet to attack different CLIP models, evaluating each of the 8 target categories. This invention utilizes 32 images to compute features for each target category.

[0152] The experimental results are shown in Table 3:

[0153]

[0154] Table 3

[0155] Table 3 presents the results of cross-modal transferable backdoor attacks on zero-shot image classification on the Caltech-101 and Food-101 datasets. Based on the results, this invention draws the following observations:

[0156] Stealth: Compared to the unattacked model, the ACC of the THNet-attacked model only decreased slightly on both datasets, and even increased in some cases. For example, on Caltech-101, the ACC of ViT-G / 14, ViT-L / 14, ViT-B / 32, and ResNet-101 decreased by 0.14, 0.14, 0.02, and -0.28, respectively. These results further demonstrate the stealth of THNet when there are no triggers in the input.

[0157] Toxicity: THNet achieved 100% ASR for all target models on both datasets, demonstrating that THNet can effectively manipulate model predictions in CLIP-based zero-shot image classification tasks.

[0158] Transferability: THNet consistently maintains high ASR across 8 target categories on four CLIP models and two datasets, with minimal impact on ACC. These results further validate THNet's transferability, demonstrating that after a single pre-training, it can be quickly transferred to any CLIP model and applied to any target category without further training.

[0159] The following is a comparison between the method of the present invention and conventional methods:

[0160] This invention employs three traditional backdoor attack methods for comparison: PBCL, BadEncoder, and BadCLIP. Since PBCL requires image-text pairs for training, this invention uses the Flickr30K dataset for fair comparison. Flickr30K contains 31,783 images, each with five text descriptions. For fair comparison, all attack methods (including THNet from this invention) are pre-trained or fine-tuned on the same dataset. When attacking the CLIP model of the ViT-L / 14 variant, this invention tests the transfer performance of all attack methods on MSCOCO. Because THNet can attack both the image and text encoders of CLIP, while the above benchmark methods can only attack the image encoder, this invention only compares the performance of attacking the image encoder in the image-to-text retrieval task. This invention reports average results across 10 target categories.

[0161] Table 4 shows a comparison of different backdoor attack transfers from Flickr30K to MSCOCO. "Attack" represents the attack method, including existing Poisoning and Backdoor Contrast Learning (PBCL), Bad Encoder, Bad CLIP, and the cross-modal transferable backdoor attack (THNet) of this invention. "PT / FT Data" represents the data type required for pre-training / fine-tuning during backdoor implantation for each attack method; "PT" indicates pre-training, and "FT" indicates fine-tuning. "-" indicates an unattacked CLIP model. Compared to existing methods, THNet performs best on ASR@1 and ACC@1, indicating stronger stealth and toxicity in transferable backdoor attacks. Furthermore, while traditional methods require several hours to attack the target class of the target model, THNet's transfer time is significantly reduced to 2.8 seconds. These results collectively demonstrate that THNet has superior performance in transferable backdoor attacks. Avg.Time represents the average attack time, Image-Text Pairs represents image-text pairs, Labeled Images represents labeled images, and Unlabeled Images represents unlabeled images.

[0162]

[0163] Table 4

[0164] The present invention also included a qualitative analysis. Figure 6This paper presents qualitative results of a backdoor attack against CLIP version ViT-B / 32 on the MSCOCO dataset. Figure 6 (a) represents the image retrieval text task, and (b) represents the text retrieval image task. For the image retrieval text task, although the attacked CLIP model can retrieve correct results with clean image queries, THNet can manipulate CLIP to retrieve text related to the target category (e.g., motorcycles, airplanes, and cats) when using image queries with triggers (st). Similarly, in the text retrieval image task, CLIP performs normally with clean text queries, but retrieves images of the target category (e.g., birds and bears) when querying text with triggers (st). These results further demonstrate the toxicity and effectiveness of THNet in transferable backdoor attacks.

[0165] The cross-modal transferable backdoor attack device provided by the present invention will be described below. The cross-modal transferable backdoor attack device described below can be referred to in correspondence with the cross-modal transferable backdoor attack method described above.

[0166] The cross-modal transferable backdoor attack device provided by this invention, with reference to Figure 7 As shown, it includes:

[0167] Dataset construction module 210 is used to obtain an unlabeled raw dataset and construct a backdoor dataset containing triggers based on the raw dataset;

[0168] The loss calculation module 220 is used to input the original dataset and the backdoor dataset containing triggers into a pre-constructed hacker network, and calculate the clean data silencing loss and the backdoor toxicity loss with triggers respectively; wherein, the clean data silencing loss corresponds to the original dataset, and the backdoor toxicity loss with triggers corresponds to the backdoor dataset containing triggers.

[0169] The pre-training module 230 is used to pre-train the pre-built hacker network by minimizing the clean data silencing loss and the backdoor poisoning loss with triggers, so as to obtain the pre-trained hacker network.

[0170] Backdoor transfer module 240 is used to calculate the feature representation of the target category through image data and text data of the target category, and to perform backdoor transfer using the pre-trained hacker network and the feature representation of the target category to generate a target CLIP model with a backdoor.

[0171] Attack module 250 is used to launch an attack using the target CLIP model with the backdoor.

[0172] Figure 8An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8 As shown, the electronic device may include a processor 310, a communications interface 320, a memory 330, and a communication bus 340. The processor 310, communications interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 can invoke logical instructions from the memory 330 to execute a cross-modal transferable backdoor attack method.

[0173] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0174] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the cross-modal transferable backdoor attack methods provided by the above methods.

[0175] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the cross-modal transferable backdoor attack methods provided by the methods described above.

[0176] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0177] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0178] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cross-modal transferable backdoor attack method, characterized in that, include: Obtain the unlabeled raw dataset, and construct a backdoor dataset containing triggers based on the raw dataset; The original dataset and the backdoor dataset containing triggers are input into a pre-constructed hacker network, and the clean data silencing loss and the backdoor toxicity loss with triggers are calculated respectively; wherein, the clean data silencing loss corresponds to the original dataset, and the backdoor toxicity loss with triggers corresponds to the backdoor dataset containing triggers. By minimizing the clean data silencing loss and the backdoor poisoning loss with triggers, the pre-built hacker network is pre-trained to obtain the pre-trained hacker network; The feature representation of the target category is calculated using image and text data of the target category. The backdoor transfer is performed using the pre-trained hacker network and the feature representation of the target category to generate a target CLIP model with a backdoor. Backdoor attacks are performed using the target CLIP model with backdoors. Image data and text data containing triggers are input into the target CLIP model with backdoors. The success rate of the attack is measured by comparing the model output before and after the attack, using accuracy, recall, and F1 score to evaluate the impact of the backdoor attack on the model behavior and for model security analysis. The original dataset includes an image dataset and a text dataset; the backdoor dataset includes an image dataset containing triggers and a text dataset containing triggers; the clean data silencing loss includes clean image silencing loss and clean text silencing loss; the backdoor poisoning loss with triggers includes backdoor image poisoning loss and backdoor text poisoning loss; and the pre-built hacking network includes a visual hacking network. and text hacking network ; Clean image silent loss And clean text silent loss for: ； In the formula, and These are the number of images and the number of text, respectively. and Clean images and clean text, respectively. and They are image datasets and text datasets, respectively; Backdoor image toxicity loss and backdoor text toxicity loss for: ； ； In the formula, and These are the backdoor image and backdoor text, obtained by analyzing the clean image. and clean text Add triggers to generate respectively and ; and These are image datasets containing triggers and text datasets containing triggers, respectively. For the preset threshold, This indicates taking the maximum value; The pre-trained hacking network includes a pre-trained visual hacking network and a pre-trained text hacking network; the step of using the pre-trained hacking network and the feature representation of the target category to perform backdoor transfer and generate a target CLIP model with a backdoor includes: Using the output of the pre-trained hacker network and the feature representation of the target category, a backdoor transfer is performed using the following formula to generate a target CLIP model with a backdoor: ； ； In the formula, For input data, and These are the image features and text features output by the target CLIP model with a backdoor, respectively. For normalization function, and These are the image encoder and text encoder for the target CLIP model, respectively. The output of the pre-trained visual hacking network, The output of the pre-trained text hacking network, This represents the features of the target category.

2. The cross-modal transferable backdoor attack method according to claim 1, characterized in that, The construction of a backdoor dataset containing triggers based on the original dataset includes: Apply an image trigger function to each image in the image dataset to generate an image dataset containing the triggers; Apply a text trigger function to each text in the text dataset to generate a text dataset containing the triggers.

3. The cross-modal transferable backdoor attack method according to claim 1, characterized in that, The hacker network outputs a feature value of zero when there is no trigger in the input data, and an output feature value greater than a preset threshold when there is a trigger in the input data.

4. The cross-modal transferable backdoor attack method according to claim 2, characterized in that, The process of pre-training the pre-built hacker network by minimizing the clean data silencing loss and the backdoor poisoning loss with triggers to obtain the pre-trained hacker network includes: The first total loss is calculated based on the clean image silencing loss and the backdoor image toxicity loss. The visual hacking network is then pre-trained based on the first total loss to obtain the pre-trained visual hacking network. The second total loss is calculated based on the clean text silencing loss and the backdoor text toxicity loss. The text hacking network is then pre-trained based on the second total loss to obtain the pre-trained text hacking network.

5. The cross-modal transferable backdoor attack method according to claim 1, characterized in that, The calculation of the feature representation of the target category using image data and text data of the target category includes: The feature representation of the target category is calculated based on the image and text data of the target category using the following formula: ； In the formula, For the feature representation of the target category, For the target category, For normalization function, and From the target category The number of images and text samples in the middle. and From the target category Image and text datasets sampled from medium. For target category Image data, For target category Text data, and These are the image encoder and text encoder for the target CLIP model, respectively. For image encoders using the target CLIP model Process target category Image data The obtained feature vector, For text encoders using the target CLIP model Process target category Text data The resulting feature vectors.

6. A cross-modal transferable backdoor attack device, characterized in that, include: The dataset construction module is used to obtain an unlabeled raw dataset and construct a backdoor dataset containing triggers based on the raw dataset. The loss calculation module is used to input the original dataset and the backdoor dataset containing triggers into a pre-constructed hacker network, and calculate the clean data silencing loss and the backdoor toxicity loss with triggers, respectively; wherein, the clean data silencing loss corresponds to the original dataset, and the backdoor toxicity loss with triggers corresponds to the backdoor dataset containing triggers. A pre-training module is used to pre-train the pre-built hacker network by minimizing the clean data silencing loss and the backdoor poisoning loss with triggers, so as to obtain the pre-trained hacker network. The backdoor transfer module is used to calculate the feature representation of the target category using image data and text data of the target category, and to perform backdoor transfer using the pre-trained hacker network and the feature representation of the target category to generate a target CLIP model with a backdoor. The attack module is used to perform backdoor attacks on the target CLIP model with backdoors. It inputs image data and text data containing triggers into the target CLIP model with backdoors, compares the model output before and after the attack, and uses accuracy, recall, and F1 score to measure the success rate of the attack, evaluate the impact of the backdoor attack on the model behavior, and is used for model security analysis. The original dataset includes an image dataset and a text dataset; the backdoor dataset includes an image dataset containing triggers and a text dataset containing triggers; the clean data silencing loss includes clean image silencing loss and clean text silencing loss; the backdoor poisoning loss with triggers includes backdoor image poisoning loss and backdoor text poisoning loss; and the pre-built hacking network includes a visual hacking network. and text hacking network ; Clean image silent loss And clean text silent loss for: ； In the formula, and These are the number of images and the number of text, respectively. and Clean images and clean text, respectively. and They are image datasets and text datasets, respectively; Backdoor image toxicity loss and backdoor text toxicity loss for: ； ； In the formula, and These are the backdoor image and backdoor text, obtained by analyzing the clean image. and clean text Add triggers to generate respectively and ; and These are image datasets containing triggers and text datasets containing triggers, respectively. For the preset threshold, This indicates taking the maximum value; The pre-trained hacking network includes a pre-trained visual hacking network and a pre-trained text hacking network; the step of using the pre-trained hacking network and the feature representation of the target category to perform backdoor transfer and generate a target CLIP model with a backdoor includes: Using the output of the pre-trained hacker network and the feature representation of the target category, a backdoor transfer is performed using the following formula to generate a target CLIP model with a backdoor: ； ； In the formula, For input data, and These are the image features and text features output by the target CLIP model with a backdoor, respectively. For normalization function, and These are the image encoder and text encoder for the target CLIP model, respectively. The output of the pre-trained visual hacking network, The output of the pre-trained text hacking network, This represents the features of the target category.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the cross-modal transferable backdoor attack method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the cross-modal transferable backdoor attack method as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the cross-modal transferable backdoor attack method as described in any one of claims 1 to 5.

Images