Multi-objective backdoor attack method and device for visual-linguistic pre-training model

Abstract

The application provides a multi-target backdoor attack method and device for a visual-language pre-training model, and belongs to the technical field of artificial intelligence security. The method comprises the following steps: constructing and training independent visual and text trigger identification networks respectively; constructing a backdoor dataset for training the identification network by embedding triggers in image and text data; training the visual and text trigger identification networks by using a cross-entropy loss function; extracting feature representations of multiple samples under each target category to obtain original clean features; calculating prototype features of the text and the image; combining the prototype features with probability distributions output by the identification network to construct a multi-target feature aggregation network; and integrating the trigger identification network and the multi-target feature aggregation network into a victim CLIP model through a one-time backdoor injection operation. The method can effectively avoid traditional defense detection based on patch or insertion type triggers.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence security technology, specifically relating to a method and apparatus for multi-target backdoor attacks against vision-language pre-trained models. The method innovatively combines covert visual and textual triggers through a framework called "Multi-Target Attack System Based on Fast Transform Network" (MRTNet), enabling simultaneous attacks on multiple target categories and rapid switching of attack targets within seconds. This invention can be widely applied to fields such as deep learning model security research, attack detection and defense mechanisms, and multimodal data processing and analysis. Background Technology

[0002] With the rapid development of deep learning technology, cross-modal learning has become a key technology driving the progress of artificial intelligence, and is widely used in tasks such as image recognition, natural language processing, and multimodal reasoning. Among them, the CLIP (Contrastive Language-Image Pre-training) model, as a representative method in this field, maps image and text information to a shared semantic space through contrastive learning, achieving remarkable results in tasks such as zero-shot classification and cross-modal retrieval. However, with the widespread application of large models such as CLIP in security-sensitive fields such as autonomous driving and medical diagnosis, the security threats they face are becoming increasingly severe, and the robustness of the models against malicious attacks appears particularly weak.

[0003] In recent years, backdoor attacks, as a highly covert security threat, have attracted widespread attention in deep learning models. Attackers inject specific "triggers" into data or models during training, causing the model to perform well under normal input, but once it encounters input containing the trigger, it will execute malicious behavior pre-set by the attacker, producing incorrect prediction results. Although some progress has been made in backdoor attack research targeting unimodal (such as image or text) models, with the rise of multimodal large models (such as CLIP, LLaVA, etc.), traditional unimodal attack methods are no longer sufficient to meet the needs of effective and covert attacks on cross-modal models. Therefore, how to design a backdoor attack method that can simultaneously affect image and text modalities and effectively transfer across modalities has become a critical challenge that urgently needs to be addressed.

[0004] Currently, research on backdoor attacks against cross-modal models such as CLIP still faces many limitations. First, most existing methods rely on image-text paired data for attack training, which not only limits the applicable scenarios but also makes the attacks less covert and easily detected by data-pair-based defense mechanisms. Second, many methods require a large number of labeled malicious samples for retraining or fine-tuning the target model, making data preparation cumbersome. Furthermore, once an attack is bound to a specific target category, it is difficult to change, lacking flexibility and adaptability. More importantly, existing methods typically only support attacks on a single target category. When attackers need to switch targets, they often need to consume significant computational resources and time to retrain the model, which is impractical in dynamic adversarial scenarios. Therefore, the research community urgently needs a new attack paradigm that can overcome these limitations. Summary of the Invention

[0005] (a) The technical problem to be solved:

[0006] (1) Existing backdoor attack methods targeting cross-modal pre-trained models such as CLIP usually require a one-time training or fine-tuning process to bind the attack to a single target category.

[0007] For scenarios requiring simultaneous attack on multiple target categories, this application proposes a covert multi-target triggering mechanism and a multi-target fast transformation network (MRTNet). This mechanism designs various syntactic style transformation structures (such as emphasis structures, relative clause structures, and existential structures) for the text modality and various amplitude-adjustable global frequency domain trigger templates (such as amplitude...) for the visual modality. , , This allows the network to generate distinctive and stealthy triggers for different target categories. The network then uses these triggers to attack multiple target categories simultaneously through a single backdoor injection, and can quickly switch targets within seconds.

[0008] (2) Regarding the technical problem of how to achieve covert and distinguishable multi-target backdoor triggering, traditional backdoor attack triggers have high visual or semantic saliency. In multi-target attack scenarios, different triggers need to be designed for each target category. How to ensure that these triggers can reliably activate the corresponding backdoor while being difficult to detect at the human perception level is a key challenge.

[0009] For the text modality, this application utilizes pre-defined syntactic style transformations to embed triggers. These transformations, while preserving the original semantic content and fluency, embed triggers based on text structure, thus corresponding to different attack target categories. For the visual modality, using amplitude-adjustable global frequency domain trigger templates, by adding additive perturbation templates of different amplitudes to the discrete cosine transform (DCT) domain of the image, various triggers that are difficult for humans to perceive but are semantically distinguishable are generated to activate different backdoor targets.

[0010] (3) Regarding the technical problem of how to achieve rapid and dynamic switching of attack target categories without retraining or modifying the core weights of the victim CLIP model, the backdoor mechanism of existing methods is deeply coupled with the model weights, and the model must be retrained to switch targets, resulting in high computational costs.

[0011] To address this, this application constructs a Multi-Object Fast Transform Network (MRTNet), which includes a trigger recognizer independent of the object CLIP encoder and a multi-object feature aggregation network. The prototype feature vector for each object category is pre-computed using the object CLIP encoder; for text, the computation... For image computing These prototypes require only a small number of clean samples from each category. During inference, the probability distribution of the recognizer's output is... With clean features from all target prototypes and current input Stacked prototype matrix Combined, through weighted aggregation ( This generates the final output embedding. When switching attack targets, only a small number of clean samples need to be collected again for the new target category, its new prototype features are calculated, and the prototype matrix in the feature aggregation network is updated. That's it. This process does not involve any modification to the victim CLIP model or its encoder weights, nor does it require retraining the trigger recognizer, so it can be completed in seconds.

[0012] (II) Technical Solution:

[0013] To achieve the above objectives, this invention provides a method and apparatus for multi-target backdoor attacks against vision-language pre-trained models. The specific technical solution includes:

[0014] A multi-target backdoor attack method targeting a vision-language cross-modal pre-trained model, comprising the following steps:

[0015] Step S1: Train the multi-target trigger recognition network: Construct and train independent visual trigger recognition networks and text trigger recognition networks respectively;

[0016] Step S2: Construct a hidden multi-target backdoor dataset: Construct a backdoor dataset for training the recognition network by embedding triggers in image and text data;

[0017] Step S3: Define and optimize the loss function of the multi-target trigger recognition network: For both visual and text trigger recognition networks, the cross-entropy loss function is used. Conduct training;

[0018] Step S4: Calculate and bind multi-target prototype features: Using the original encoder of the victim CLIP model, extract feature representations of multiple image and text samples under each target category to obtain the original clean features; for the text target category, calculate its text prototype features. For visual target categories, calculate their image prototype features. These prototype features are combined with the probability distribution of the recognition network output to construct a multi-target feature aggregation network. This network uses the output probability of the recognition network as weights to perform weighted aggregation of multiple target prototype features and original clean features that fuse image and text prototype features, generating the final output embedding. ;

[0019] Step S5: Implement a backdoor attack with fast target category switching: Through a single backdoor injection operation, the trained trigger recognition network and multi-target feature aggregation network are integrated into the victim CLIP model to form a backdoor model that can attack multiple target categories.

[0020] Step S6: Verify the effectiveness and stealth of multi-target backdoor attacks: Verify the attack effect of the backdoor model through downstream tasks such as cross-modal retrieval and zero-shot classification.

[0021] A multi-target backdoor attack device targeting a vision-language cross-modal pre-trained model includes:

[0022] The training module is used to train the multi-object trigger recognition network: it constructs and trains independent visual trigger recognition networks and text trigger recognition networks respectively;

[0023] The building blocks are used to construct a hidden multi-target backdoor dataset: by embedding triggers in image and text data, a backdoor dataset is constructed for training the recognition network.

[0024] The loss function definition module is used to define and optimize the loss function of a multi-object trigger recognition network: for visual and text trigger recognition networks, the cross-entropy loss function is used. Conduct training;

[0025] The computation module is used to compute and bind multi-target prototype features: using the original encoder of the victim CLIP model, it extracts feature representations of multiple image and text samples under each target category to obtain the original clean features; for the text target category, it computes its text prototype features. For visual target categories, calculate their image prototype features. These prototype features are combined with the probability distribution of the recognition network output to construct a multi-target feature aggregation network. This network uses the output probability of the recognition network as weights to perform weighted aggregation of multiple target prototypes and original clean features, generating the final output embedding. ;

[0026] The attack module is used to implement backdoor attacks that quickly switch target categories: through a single backdoor injection operation, the trained trigger recognition network and multi-target feature aggregation network are integrated into the victim CLIP model to form a backdoor model that can attack multiple target categories.

[0027] The verification module is used to verify the effectiveness and stealth of multi-target backdoor attacks: the attack effect of the backdoor model is verified through downstream tasks such as cross-modal retrieval and zero-shot classification.

[0028] A computing device includes: at least one processor and a memory storing program instructions; when the program instructions are read and executed by the processor, the computing device performs the method.

[0029] A readable storage medium storing program instructions that, when read and executed by a computing device, cause the computing device to perform the method.

[0030] A computer program product includes a computer program that, when executed by a processor, implements the method.

[0031] The beneficial effects of this invention are:

[0032] This invention provides an innovative cross-modal backdoor attack framework that supports rapid switching between multiple targets. By designing a covert multi-target triggering mechanism (such as utilizing text syntactic style transfer and image frequency domain amplitude perturbation) and a decoupled multi-target fast transformation network, this method can simultaneously bind and attack multiple target categories in a single backdoor implantation, achieving target category switching within seconds without retraining the victim model. This method demonstrates high attack success rate and strong stealth in various CLIP model architectures and downstream tasks, with minimal impact on normal model function, and effectively circumvents traditional defense detection based on patches or insertion triggers. Attached Figure Description

[0033] Figure 1This is the overall architecture diagram of the covert multi-target triggering mechanism, which details how to generate covert triggers to activate different backdoor targets;

[0034] Figure 2 This is an overview diagram of the pre-training of the multimodal trigger recognizer, which shows the process of training the independent trigger recognizer in the first stage of MRTNet.

[0035] Figure 3 This is an overview diagram of few-shot backdoor transfer and feature aggregation, which illustrates the working mechanism of the second stage of MRTNet, namely the multi-target feature aggregation network.

[0036] Figure 4 The results are qualitative findings of a transferable backdoor attack on cross-modal retrieval on the MSCOCO dataset, where (a) shows the results of an example of image-to-text retrieval and (b) shows the results of an example of text-to-image retrieval.

[0037] Figure 5 This is a quantitative performance comparison of the MRTNet attack on different CLIP models. The figure uses a bar chart to visually compare the quantitative performance of the MRTNet attack on four different CLIP architectures. Detailed Implementation

[0038] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0039] This invention provides a multi-target backdoor attack method targeting a vision-language cross-modal pre-trained model. The multi-target backdoor attack method is based on visual and text triggers and employs a MultipleTargetClasses by Rapid Transformation Network (MRTNet) model. The method comprises two key stages: generating a hidden multi-target trigger mechanism and constructing the MRTNet.

[0040] Figure 1 This is an overall architecture diagram of a stealthy multi-target triggering mechanism, detailing how covert triggers are generated to activate different backdoor targets. For text modalities, such as... Figure 1 As shown on the left, various preset syntactic style transformation structures are applied to clean raw text to embed text-structure-based triggers. These transformations preserve the original semantics and fluency to the greatest extent possible, while using specific sentence variations to identify different attack targets. Figure 1 Three typical syntactic transformation patterns are demonstrated: emphatic structures, relative clause structures, and existential structures. Each transformation pattern independently generates a poisoned text variant, corresponding to a specific attack target category. For visual modalities, such as... Figure 1 As shown on the right, the triggers are constructed in the frequency domain to enhance stealth. The process is as follows: First, a discrete cosine transform (DCT) is performed on the clean original image to convert it from the spatial domain to the frequency domain representation. Then, a fixed global perturbation template is added to the frequency domain, and different amplitude factors are multiplied to generate a poisoned frequency domain representation with adjustable intensity; different amplitudes correspond to different attack targets. Finally, the poisoned frequency domain representation is converted back to the spatial domain using inverse discrete cosine transform (IDCT), resulting in a visually imperceptible poisoned image. This method of applying global, weak perturbations in the frequency domain is more stealthy than the traditional method of adding obvious local patches in the spatial domain. The poisoned dataset containing multiple trigger modes generated based on the above mechanism lays the foundation for subsequent training of a multimodal recognizer capable of accurately identifying and distinguishing different triggers.

[0041] The method includes the following steps:

[0042] Step S1: Train the multi-target trigger recognition network. Construct and train separate visual trigger recognition networks and text trigger recognition networks. For image data, use a pre-trained ViT (Vision Transformer) as the infrastructure; for text data, use the BERT model as the infrastructure. The goal of the recognition network is to accurately identify the trigger category (corresponding to different attack targets) when the input data contains specific triggers; when the input is clean data, it is identified as "clean," thus ensuring no impact on normal data.

[0043] Figure 2 This is an overview diagram of the pre-training of the multimodal trigger recognizer, illustrating the process of training independent trigger recognizers in the first stage of MRTNet. After preparing poisoned data containing multiple trigger modes, independent trigger recognizers are trained separately for text and visual modalities. The text recognizer is based on the BERT architecture, and the visual recognizer is based on the VisionTransformer (ViT) architecture. The goal of the recognizer is to accurately determine whether the input contains triggers and the specific type of trigger (corresponding to k target categories), or to classify the input as a "clean" category. Figure 2The left half of the diagram details the training process of the visual trigger recognizer: a clean original image is perturbed in the frequency domain to generate poisoned images corresponding to different target categories (the first category to the kth category); these images are input into the recognizer of the ViT architecture, and the recognizer outputs a probability distribution (the example in the diagram predicts a probability of 0.94 for the "clean" category); this prediction result is trained under supervision using cross-entropy loss, forcing the recognizer to learn to distinguish frequency domain triggers of different amplitudes. Figure 2 The right half symmetrically illustrates the training process of the text trigger recognizer: clean original text undergoes different syntactic style transformations to generate poisoned text variants corresponding to different target categories; these texts are input into the BERT-based recognizer, which also outputs a probability distribution (the example in the figure predicts a probability of 0.96 for the "clean" category), and is trained using cross-entropy loss to reliably recognize different syntactic trigger structures. Formally, for the input... The recognizer outputs a probability distribution. ,in This represents the classification head of the classifier. The classifier uses standard cross-entropy loss. Train it to reliably distinguish between various covert triggering patterns and clean inputs.

[0044] Step S2: Construct a concealed multi-target backdoor dataset. A backdoor dataset for training the recognition network is constructed by embedding a series of imperceptible triggers into image and text data. Specifically, for image data, a global frequency domain perturbation method is used. By adjusting additive perturbation templates of different amplitudes in the Discrete Cosine Transform (DCT) domain, various visual triggers are generated, forming a contaminated image dataset. For text data, a syntactic style conversion method is used to apply predefined transformations such as emphasis structures, relative clause structures, or existential structures to the original sentences. While preserving semantics, text triggers are embedded to form a contaminated text dataset. ).

[0045] Unlike existing backdoor attack methods based on fixed patches, the MRTNet of this invention aims to generate imperceptible triggering patterns to attack multiple target categories simultaneously. For the text modality, this is achieved by processing clean text... Predefined syntactic style transfers are applied to generate poisoned text, such as using emphatic structures ("It is...that"), relative clause structures (using "that is", "which has", etc.), and existential structures ("There is / are"). Each style transfer... Generate a poison variant independently This allows for the construction of a poisoned text set containing multiple triggering patterns for each clean input. For the visual modality, a trigger is constructed in the frequency domain. For a clean image... Performing discrete cosine transform yields By adding a fixed perturbation template And multiplied by different amplitude factors To generate a poisoned frequency domain representation Then, through inverse transformation, a visually imperceptible poisoning image is obtained. By setting multiple amplitudes It can generate image trigger sets corresponding to different target categories.

[0046] Step S3: Define and optimize the loss function of the multi-target trigger recognition network. For both visual and text trigger recognition networks, the cross-entropy loss function is used. The network is trained using a loss function designed to enable the recognition network to correctly classify inputs with different triggers into their corresponding trigger categories (or target categories), and to classify clean inputs into the "clean" category, thereby achieving accurate trigger detection.

[0047] Step S4: Calculate and bind multi-target feature prototypes. Using the original encoder of the victim CLIP model, extract feature representations from multiple (e.g., 32) image and text samples for each target category to obtain the original clean features. For the text target category, calculate its text prototype features ( For visual target categories, calculate their image prototype features ( ); These prototype features are combined with the probability distribution of the recognition network output to construct a multi-target feature aggregation network. This network uses the output probability of the recognition network as weights to perform weighted aggregation of multiple target prototype features and original clean features that fuse image and text prototype features, generating the final output embedding. ).

[0048] Figure 3 This is an overview diagram of few-shot backdoor transfer and feature aggregation. The diagram illustrates the working mechanism of the second stage of MRTNet, namely the multi-target feature aggregation network. The core of this method lies in binding features from multiple target categories to the victim CLIP model through a single backdoor injection. , It supports fast switching. The process does not require retraining or fine-tuning the CLIP encoder itself. First, for each target category... Pre-compute its prototype features. For text targets, text prototype features... This is obtained by averaging the features output by the CLIP text encoder for all clean text samples in this category; for visual targets, the image prototype features are... This feature is obtained by averaging the features output from the CLIP image encoder for all clean image samples of this category. Next, a multi-objective feature aggregation network is constructed. For any input... (Image or text), first extract its features using the original CLIP encoder. Next, we will perform matrix merging. Individual target prototype characteristics and Stack them row by row to form a prototype matrix. Each target prototype feature It is a cross-modal fusion feature, composed of image prototype features corresponding to the target category. and text prototype features They are calculated and fused together, jointly encoding the visual and semantic information of this category. It is the feature dimension.

[0049] Meanwhile, the recognition network outputs a probability vector. ,in It is the first Each output probability. The multi-target feature aggregation network obtains the final output features through weighted aggregation. : This design ensures the stealth and toxicity of the attack: when the input is clean data, the identifier will detect it with a high probability. Assign to the "clean" category and output The model behaves normally. When the input contains a specific trigger, the recognizer will assign it with a high probability to the corresponding target category. This makes the output The features were successfully directed to the attacker's pre-set target.

[0050] This method employs a few-sample approach to compute the target class feature representation, enabling efficient backdoor transfer and embedding the backdoor into the target CLIP model. This results in attacks with high cross-modal transferability and stealth. Specifically, the method rapidly computes the prototype feature vector by averaging the CLIP encoder output of a small number of clean samples from each target class. )and( These few-sample prototype features are combined with the probability distribution of the trigger recognizer output, and then weighted and aggregated using a formula ( ). )(in (The target prototype feature matrix) efficiently transfers the model's output features to the attacker's preset target category without retraining the victim model, ensuring the stealth of the attack and efficient cross-modal transferability.

[0051] By introducing an attack mechanism based on multi-target prototype feature aggregation, it is ensured that on data containing triggers, the model's output features can be efficiently and accurately guided to the attacker's preset target category.

[0052] Step S5: Implement a backdoor attack with rapid target category switching. Through a single backdoor injection operation, the trained trigger recognition network and multi-target feature aggregation network are integrated into the victim CLIP model, forming a backdoor model capable of attacking multiple target categories. When switching attack targets, there is no need to retrain the model or recognition network; only the features of a small number of clean samples need to be extracted for the new target category, and its new prototype features calculated. And update the prototype matrix in the multi-objective feature aggregation network. This allows for target switching within seconds.

[0053] Rapid target category switching is a key advantage of this method. When it's necessary to switch attack targets from a set... (in (Indicates switching from the first target category to the kth target category) to a new set. At that time, it is only necessary to calculate the prototype features for the new target class. and use them to update the prototype matrix. The corresponding rows in the middle form The parameters of both the trigger recognizer and the CLIP encoder remain unchanged. This switching process involves feature extraction and averaging of only a small number of samples, which can be completed within seconds, enabling dynamic and efficient switching of backdoor attack targets. Through this implementation, MRTNet can efficiently and covertly carry out multi-target backdoor attacks on the CLIP model, while ensuring the transferability and rapid switching capability of the attack across multiple modalities.

[0054] Step S6: Verify the effectiveness and stealth of the multi-target backdoor attack. The attack effect of the backdoor model is verified through downstream tasks such as cross-modal retrieval and zero-shot classification. Evaluation metrics include: Attack Success Rate (ASR), which is the proportion of inputs with triggers that are successfully misled to the target class; and Clean Data Accuracy (ACC), which is the degree to which the model retains its performance when processing normal inputs. Experiments show that the method achieves an ASR exceeding 97.7% while exhibiting minimal ACC decline, and its style- and frequency-domain-based triggers effectively resist detection by existing defense methods.

[0055] Figure 4 This is a qualitative result of a transferable backdoor attack detected across modalities on the MSCOCO dataset. The figure visually demonstrates the effectiveness and multi-target capabilities of MRTNet through examples. Figure 4 In the image-to-text retrieval example in (a), the left side shows a clean query image and its normal retrieval results. The right side shows, side by side, three different malicious retrieval results triggered by MRTNet when the same query image is embedded with three different amplitude frequency domain visual triggers. Specifically: when the embedding amplitude... When the trigger was activated, the search results were successfully misled to the attacker's pre-defined "train" target category. Figure 4 In the text-to-image retrieval example in (b), the left side shows the clean query text and its normal retrieval results. The right side shows the different malicious retrieval results triggered when the same query text is embedded with text triggers of three different syntactic styles. These vivid qualitative results demonstrate that MRTNet can reliably and accurately direct cross-modal retrieval results to multiple arbitrarily preset attack targets without affecting the model's normal response to clean input, through multiple covert and distinguishable triggering modes. This fully verifies the powerful capability of the proposed method in achieving efficient and covert multi-target backdoor attacks.

[0056] Figure 5 This chart presents a quantitative performance comparison of the MRTNet attack on different CLIP models. The bar chart visually compares the quantitative performance of the MRTNet attack on four different CLIP model architectures. The overall results demonstrate that the MRTNet attack framework can achieve both extremely high attack success rates and extremely low normal performance loss across multiple CLIP model architectures, fully validating its cross-model portability and its seriousness as a real-world threat.

[0057] Implementation results:

[0058] Dataset:

[0059] (1) ImageNet: Contains 1,281,167 training images and 50,000 validation images, covering 1,000 categories. In this method, its unlabeled image data is used to pre-train a visual trigger recognizer for generating and recognizing multi-target visual triggers based on frequency domain perturbations.

[0060] (2) MSCOCO: Contains over 120,000 images, each accompanied by 5 text descriptions. This dataset is an authoritative benchmark for cross-modal retrieval tasks. Its image-text pair data is used to generate text poisoning samples based on syntactic style transfer to train a text trigger recognizer, and also serves as the primary test set to evaluate the attack performance and stealth of MRTNet on cross-modal retrieval tasks.

[0061] Target Models: To verify the broad applicability and transferability of the proposed method to CLIP models with different architectures, four representative CLIP models were selected as victim models for attack testing. These models are distinguished by their image encoder architecture, including: ViT-G / 14, ViT-L / 14, and ViT-B / 32 based on Transformer, and ResNet-101 based on convolutional neural network.

[0062] Evaluation metrics: Accuracy (ACC) and Attack Success Rate (ASR) are used as core evaluation metrics to comprehensively measure the performance of backdoor attacks.

[0063] Accuracy (ACC): Measures the cross-modal retrieval performance of a backdoored model when processing clean (trigger-free) inputs (such as image-to-text I2T or text-to-image T2I retrieval). A high ACC value indicates that the backdoor attack has minimal impact on the normal functioning of the model, thus ensuring the stealth of the attack.

[0064] Attack Success Rate (ASR): Measures the proportion of times a model's output is successfully misled to a pre-defined target category when a specific trigger is included in the input. A high ASR value directly reflects the effectiveness (toxicity) of a backdoor attack. In the experiments, the ACC@K and ASR@K values ​​for Top-1, Top-5, and Top-10 search results are reported.

[0065] Implementation details: The multi-target backdoor attack framework MRTNet includes two independent trigger recognition systems: visual and textual.

[0066] Visual trigger recognizer: Employs the Vision Transformer (ViT) as its architecture. This is achieved by adding elements with different amplitudes to the frequency domain representation of ImageNet images. Global perturbation template A poisoned image dataset corresponding to three target categories was synthesized and used to train the recognizer.

[0067] Text trigger recognizer: Employs the BERT-based model as its foundation. By applying three predefined syntactic style transformations (emphasis structure, relative clause structure, and existential structure) to clean text in the MSCOCO dataset, corresponding poisoned text variants are generated and used to train the recognizer.

[0068] Multi-target binding and switching: During the attack phase, the average features of clean samples in the CLIP model are calculated as prototypes for each target category. The output of the trigger recognizer is bound to these prototype features through the proposed multi-target feature aggregation network to achieve backdoor functionality. When switching attack targets, only the prototype features for the target category need to be recalculated and the binding relationships updated, without retraining the victim CLIP model or trigger recognizer, achieving second-level fast switching.

[0069] Cross-modal search results:

[0070] The performance of the proposed MRTNet in downstream cross-modal retrieval tasks was evaluated, based on cross-modal semantic similarity computed using CLIP.

[0071] Two widely used cross-modal retrieval benchmark datasets were used for evaluation:

[0072] (1) MSCOCO: Contains 123,287 images, each with 5 manually annotated text descriptions, covering 80 common object categories. This dataset is the authoritative benchmark for visual-language tasks.

[0073] (2) FashionGen: Provides images and text descriptions of 48 categories of fashion products (such as clothing and accessories) to evaluate the effectiveness of attacks in specific vertical fields.

[0074] Accuracy (ACC@K) and attack success rate (ASR@K) are used as core metrics. ACC@K measures the proportion of true results appearing in the top K search results when the model uses clean queries, reflecting the stealth of backdoor attacks. ASR@K measures the proportion of results returned by the model when the query contains triggers, reflecting the toxicity of the attack. Retrieval results in both image-to-text (I2T) and text-to-image (T2I) directions are reported (K=1,5,10). In experiments, a pre-trained MRTNet was used to launch attacks on three target categories simultaneously, and the average results were taken across 10 different target category configurations.

[0075] Experimental results:

[0076] Table 1 shows the results of a transferable backdoor attack on cross-modal retrieval on the MSCOCO dataset. This table compares the performance of different CLIP models (ViT-G / 14, ViT-L / 14, etc.) on image-to-text (I2T) and text-to-image (T2I) retrieval tasks before and after the MRTNet attack. Table 2 shows the results of a transferable backdoor attack on cross-modal retrieval on the FashionGen dataset. Experiments on another cross-modal retrieval dataset, FashionGen, further validated the generalization ability of MRTNet.

[0077] Table 1. Results of transferable backdoor attacks on cross-modal retrieval on the MSCOCO dataset.

[0078]

[0079] Table 2. Results of transferable backdoor attacks on cross-modal retrieval on the FashionGen dataset.

[0080]

[0081] Based on the experimental results, the following conclusions were drawn:

[0082] Stealth: Despite the extremely potent attack, MRTNet has minimal impact on the normal functioning of the model. Compared to the unattacked original model, the attacked model's retrieval accuracy (ACC@K) drops only slightly. For example, on MSCOCO, the performance difference is typically no more than 4.9 percentage points; on FashionGen, the difference is typically less than 2.0 percentage points, and in some cases, the ACC even slightly improves. This indicates that without trigger input, the model behaves almost identically to the normal model, ensuring the stealth of the attack.

[0083] Toxicity: MRTNet demonstrates near-perfect attack success rates. On the MSCOCO dataset, all victimized models achieved ASR@1 exceeding 99.7% on the I2T task and over 99.2% on the T2I task. On the FashionGen dataset, ASR@1 reached 99.9% or even 100% in most cases. This proves that once the trigger is activated, MRTNet can almost completely manipulate the model's output, directing it towards multiple pre-defined target categories.

[0084] Transferability: After a single pre-training, MRTNet can successfully transfer and efficiently attack four CLIP models (including Transformer and CNN classes) with vastly different architectures without requiring retraining or fine-tuning for specific victim models. Furthermore, it maintains consistently high attack success rates and stealth on two different domain datasets (general object MSCOCO and vertical domain FashionGen) and two retrieval directions (I2T and T2I). This fully demonstrates the powerful generalization and transferability of the MRTNet framework across models, tasks, and datasets.

[0085] Zero-shot image classification results:

[0086] The following evaluation assesses MRTNet's performance on zero-shot image classification tasks, based on the semantic similarity between images and category descriptions, following the CLIP standard protocol.

[0087] Zero-shot image classification was performed using two widely used datasets:

[0088] (1) ImageNet contains images of 1,000 object classes, and its training set has 1,281,167 unlabeled images.

[0089] (2) MSCOCO contains images of 80 common objects.

[0090] Evaluation was performed on the MSCOCO dataset. Four different CLIP models (ViT-G / 14, ViT-L / 14, ViT-B / 32, and ResNet-101) were attacked using the pre-trained MRTNet framework. In the experiments, attacks were launched simultaneously against three target classes, and their average performance was calculated.

[0091] Experimental results:

[0092] Figure 5 The experimental results for zero-shot image classification are shown. We can observe that:

[0093] Stealth: Compared to the original CLIP model, the model attacked by MRTNet shows only a very small decrease in zero-shot classification accuracy. For example, for ViT-G / 14, ViT-L / 14, ViT-B / 32, and ResNet-101 models, the accuracy drops by only 2.19%, 0.91%, 0.15%, and 0.03%, respectively. This tiny performance loss is difficult to detect in practical scenarios, further demonstrating the high stealth of the MRTNet attack.

[0094] Toxicity: While maintaining high stealth, MRTNet achieved an attack success rate (ASR) of over 97.7% against all four target models. This demonstrates that, within a zero-shot classification framework, MRTNet can reliably misclassify images with triggers into multiple target categories pre-defined by the attacker.

[0095] Transferability: This experiment further confirms the transferability of MRTNet. A visual trigger recognizer trained on ImageNet (a different dataset from the downstream task MSCOCO) can successfully transfer and effectively attack CLIP's zero-shot classification capabilities without affecting its normal classification performance. This demonstrates that the threat posed by MRTNet is pervasive and can penetrate various downstream application systems that rely on the CLIP encoder.

[0096] Comparison with traditional methods:

[0097] To comprehensively evaluate the state-of-the-art performance of MRTNet, it was compared with three mainstream backdoor attack benchmark methods: PBCL, BadEncoder, and BadCLIP. To ensure fairness, since methods like PBCL require image-to-text data for training, all methods were pre-trained or fine-tuned using the Flickr30K dataset (containing 31,783 images, each with 5 text descriptions). Subsequently, the transferability of these methods after attacking the ViT-L / 14 architecture CLIP model was tested on the MSCOCO dataset. Given that PBCL, BadEncoder, and BadCLIP can only attack image encoders, for fairness, the comparison was limited to image-to-text (I2T) retrieval tasks. Average results across multiple target categories are reported.

[0098] The experimental results are shown in Table 3, which compares the efficiency of target category switching. The core of this table is to demonstrate MRTNet's revolutionary rapid switching capability. In terms of attack performance, MRTNet achieved a near-perfect attack success rate (ASR@1=99.93%), while maintaining a clean data accuracy similar to the baseline model (ACC@1=36.44%). This proves that MRTNet surpasses existing methods in both attack toxicity and attack stealth.

[0099] Table 3 compares the efficiency of target category switching.

[0100]

[0101] However, MRTNet's most significant advantage lies in its unprecedented target switching efficiency. Traditional backdoor attack methods (such as PBCL, BadEncoder, and BadCLIP) follow a "train once, bind one target" model. When attacking multiple target categories, the victim model must be retrained or fine-tuned multiple times, which is time-consuming. As shown in Table 3, to cover 9 target categories, PBCL, BadEncoder, and BadCLIP require a total of approximately 117,715.3 seconds, 217,334.7 seconds, and 109,778.8 seconds, respectively. In contrast, MRTNet, with its decoupled architecture, only needs to extract features from a small number of clean samples and update the prototype binding relationship after the initial injection to achieve target switching. It only takes 25.4 seconds to complete the switching of all 9 categories, with an average switching time of only 8.6 seconds per category. This means that MRTNet reduces the switching time cost of multi-target attacks to less than 0.1% of that of traditional methods. This "second-level switching" capability greatly expands the threat potential and flexibility of backdoor attacks in real-world scenarios, enabling them to quickly adapt to dynamically changing attack requirements.

[0102] Qualitative analysis:

[0103] Figure 5 This paper visually demonstrates the qualitative effect of MRTNet in carrying out a multi-target backdoor attack on the CLIP model (taking ViT-B / 32 as an example) on the MSCOCO dataset. This attack reveals that AI models can be implanted with hidden backdoors without disrupting normal functions. Once the triggering conditions are met, the output can be maliciously manipulated. This threat is particularly serious in security-critical fields such as autonomous driving and content moderation.

[0104] These qualitative results vividly demonstrate that MRTNet can reliably direct cross-modal retrieval results to multiple arbitrary attack targets (toxicity) through various imperceptible triggering modes without affecting the normal function of the model (stealth), fully demonstrating its powerful capabilities and real-world threat as a multi-target, fast-switching backdoor attack framework.

[0105] This invention also provides a multi-target backdoor attack device targeting a vision-language pre-trained model, used to implement the above methods, comprising:

[0106] The training module is used to train the multi-object trigger recognition network: separate visual trigger recognition networks and text trigger recognition networks are built and trained; for image data, the pre-trained ViT model is used as the basic architecture; for text data, the BERT model is used as the basic architecture.

[0107] The building blocks are used to construct a hidden multi-target backdoor dataset: by embedding triggers in image and text data, a backdoor dataset is constructed for training the recognition network.

[0108] The loss function definition module is used to define and optimize the loss function of a multi-object trigger recognition network: for visual and text trigger recognition networks, the cross-entropy loss function is used. Conduct training;

[0109] The computation module is used to compute and bind multi-target prototype features: using the original encoder of the victim CLIP model, it extracts feature representations of multiple image and text samples under each target category to obtain the original clean features; for the text target category, it computes its text prototype features. For visual target categories, calculate their image prototype features. These prototype features are combined with the probability distribution of the recognition network output to construct a multi-target feature aggregation network. This network uses the output probability of the recognition network as weights to perform weighted aggregation of multiple target prototypes and original clean features, generating the final output embedding. ;

[0110] The attack module is used to implement backdoor attacks that quickly switch target categories: through a single backdoor injection operation, the trained trigger recognition network and multi-target feature aggregation network are integrated into the victim CLIP model to form a backdoor model that can attack multiple target categories.

[0111] The verification module is used to verify the effectiveness and stealth of multi-target backdoor attacks: the attack effect of the backdoor model is verified through downstream tasks such as cross-modal retrieval and zero-shot classification.

[0112] The present invention also provides a computing device, comprising: at least one processor and a memory storing program instructions; when the program instructions are read and executed by the processor, the computing device performs the method described thereon.

[0113] The present invention also provides a readable storage medium storing program instructions, which, when read and executed by a computing device, cause the computing device to perform the method described thereon.

[0114] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the method described herein.

[0115] Although the invention has been described with respect to a limited number of embodiments, those skilled in the art will understand from the foregoing description that other embodiments are conceivable within the scope of the invention described herein. Furthermore, it should be noted that the language used in this specification has been chosen primarily for readability and instructional purposes, and not for the purpose of explaining or limiting the subject matter of the invention.

Claims

1. A multi-target backdoor attack method targeting a vision-language pre-trained model, characterized in that, The method includes the following steps: Step S1: Train the multi-target trigger recognition network: Construct and train independent visual trigger recognition networks and text trigger recognition networks respectively; Step S2: Construct a hidden multi-target backdoor dataset: Construct a backdoor dataset for training the recognition network by embedding triggers in image and text data; Step S3: Define and optimize the loss function of the multi-object trigger recognition network: For both visual trigger recognition and text trigger recognition networks, the cross-entropy loss function is used. Conduct training; Step S4: Calculate and bind multi-target prototype features: Using the original encoder of the victim CLIP model, extract feature representations of multiple image and text samples under each target category to obtain the original clean features; for the text target category, calculate its text prototype features. For visual target categories, calculate their image prototype features. These prototype features are combined with the probability distribution of the recognition network output to construct a multi-target feature aggregation network. This network uses the output probability of the recognition network as weights to perform weighted aggregation of multiple target prototype features and original clean features that fuse image prototype features and text prototype features, generating the final output embedding. ; Step S5: Implement a backdoor attack with fast target category switching: Through a single backdoor injection operation, the trained trigger recognition network and multi-target feature aggregation network are integrated into the victim CLIP model to form a backdoor model that can attack multiple target categories. Step S6: Verify the effectiveness and stealth of multi-target backdoor attacks: Verify the attack effect of the backdoor model through downstream tasks such as cross-modal retrieval and zero-shot classification.

2. The multi-target backdoor attack method for vision-language pre-trained models according to claim 1, characterized in that, In step S2, for the image data, a global frequency domain perturbation method is used. By adjusting additive perturbation templates of different amplitudes in the discrete cosine transform domain, multiple visual triggers are generated to form a contaminated image dataset. For text data, a syntactic style conversion method is used to apply predefined emphasis structures, relative clause structures, or existential structures to the original sentences, embedding text triggers while preserving semantics to form a contaminated text dataset. .

3. The multi-target backdoor attack method for vision-language pre-trained models according to claim 1, characterized in that, Step S4 includes: For text targets, text prototype features This is obtained by averaging the features output by the CLIP text encoder for all clean text samples in this category; for visual targets, the image prototype features are... It is obtained by averaging the features output by the CLIP image encoder for all clean image samples of this category.

4. The multi-target backdoor attack method for vision-language pre-trained models according to claim 3, characterized in that, In step S4, constructing the multi-target feature aggregation network includes: for any input First, its features are extracted using the original CLIP encoder. Then A target prototype feature that fuses image prototype features and text prototype features. Stack them row by row to form a prototype matrix. ,in It is the feature dimension, k is the number of target categories, and each target prototype feature is... It is a cross-modal fusion feature, composed of image prototype features corresponding to the target category. and text prototype features The probability vector is calculated and fused together, jointly encoding the visual and semantic information of this category; the probability vector output by the multi-object trigger recognition network is... ,in It is the first Each output probability; the multi-target feature aggregation network obtains the final output features through weighted aggregation. : .

5. The multi-target backdoor attack method for vision-language pre-trained models according to claim 1, characterized in that, In step S6, the evaluation metrics include: Attack Success Rate (ASR), which is the proportion of inputs with triggers that are successfully misled to the target category; and Clean Data Accuracy (ACC), which is the degree to which the backdoor model retains its performance when processing normal inputs.

6. The multi-target backdoor attack method for vision-language pre-trained models according to claim 1, characterized in that, Also includes: When switching attack targets, there is no need to retrain the victim CLIP model or the multi-target trigger recognition network; instead, features of multiple samples are re-extracted for the new target category, and its new prototype features are calculated. And update the prototype matrix in the multi-objective feature aggregation network. .

7. A multi-target backdoor attack device targeting a vision-language pre-trained model, characterized in that, include: The training module is used to train the multi-object trigger recognition network: it constructs and trains independent visual trigger recognition networks and text trigger recognition networks respectively; The building blocks are used to construct a hidden multi-target backdoor dataset: by embedding triggers in image and text data, a backdoor dataset is constructed for training the recognition network. The loss function definition module is used to define and optimize the loss function of a multi-object trigger recognition network: for visual and text trigger recognition networks, the cross-entropy loss function is used. Conduct training; The computation module is used to compute and bind multi-target prototype features: using the original encoder of the victim CLIP model, it extracts feature representations of multiple image and text samples under each target category to obtain the original clean features; for the text target category, it computes its text prototype features. ; For visual object categories, calculate their image prototype features. These prototype features are combined with the probability distribution of the recognition network output to construct a multi-target feature aggregation network. This network uses the output probability of the recognition network as weights to perform weighted aggregation of multiple target prototype features and original clean features that fuse image prototype features and text prototype features, generating the final output embedding. ; The attack module is used to implement backdoor attacks that quickly switch target categories: through a single backdoor injection operation, the trained trigger recognition network and multi-target feature aggregation network are integrated into the victim CLIP model to form a backdoor model that can attack multiple target categories. The verification module is used to verify the effectiveness and stealth of multi-target backdoor attacks: the attack effect of the backdoor model is verified through downstream tasks such as cross-modal retrieval and zero-shot classification.

8. A computing device, characterized in that, include: At least one processor and a memory storing program instructions; When the program instructions are read and executed by the processor, the computing device performs the multi-target backdoor attack method against the vision-language pre-trained model as described in any one of claims 1-6.

9. A readable storage medium storing program instructions, characterized in that, When the program instructions are read and executed by the computing device, the computing device performs the multi-target backdoor attack method against the vision-language pre-trained model as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the multi-target backdoor attack method against the vision-language pre-trained model as described in any one of claims 1-6.

Images