Multimodal semantic camouflage backdoor attack method based on natural attributes
By generating global environmental semantic triggers with natural attributes and cross-modal contrastive learning through deep information fusion, the problems of concealment, robustness and zero-sample transfer in backdoor attacks in models such as CLIP are solved, and efficient multimodal attack effects are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHONGQING UNIV OF POSTS & TELECOMM
- Filing Date
- 2026-03-04
- Publication Date
- 2026-06-02
AI Technical Summary
Existing backdoor attack methods suffer from poor visual concealment, cross-modal semantic inconsistency, insufficient robustness, and failure in zero-sample scenarios in large-scale multimodal models such as CLIP, resulting in low attack success rates and easy interception by defense algorithms.
A multimodal semantic spoofing backdoor attack method based on natural attributes is adopted. By generating global environmental semantic triggers and fusing deep information, combined with a visual-language alignment model for cross-modal comparative learning, a physically realistic poisoned image is generated and consistent descriptive words are embedded in the text to establish an abnormal association between the trigger and the target category.
It achieves extremely high stealth and naturalness, improves the robustness of attacks and cross-modal collaborative effects, has strong zero-sample transfer capability, can stably activate backdoors in multimodal application scenarios, and has minimal impact on the original task.
Smart Images
Figure CN122133146A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision, natural language processing and artificial intelligence security technology, and relates to a multimodal semantic spoofing backdoor attack method for vision-language alignment models (such as CLIP) and visual self-supervised learning models. Background Technology
[0002] In recent years, artificial intelligence technologies, represented by self-supervised learning and large-scale vision-language pre-training (VLP), have made groundbreaking progress. Among them, contrastive language-image pre-trained models (such as CLIP) achieve deep alignment between visual features and text semantics by performing contrastive learning on hundreds of millions of image-text pairs. This multimodal model not only performs well in image retrieval and automatic annotation, but also, with its powerful zero-shot transfer capability, has become the foundation of current artificial general intelligence (AGI). Since these models are usually trained by directly scraping massive amounts of unlabeled data from the Internet, their security issues, especially the defense against backdoor attacks during the pre-training stage, have become a research hotspot in the field of artificial intelligence security.
[0003] Existing backdoor attack schemes targeting visual models mainly include: 1) Local sticker-based attacks: such as BadNets, which use fixed pixel blocks (e.g., black and white squares) embedded at specific locations in the image as triggers. 2) Self-supervised attacks based on feature coupling: attacks targeting image self-supervised models, which use image transformations or local patches to establish associations with the target class. 3) Multimodal attacks based on specific text triggers: inserting special characters or rare words into the text and embedding corresponding digital watermarks or perturbations into the image. These schemes attempt to establish an unusual mapping between trigger features and target label features within the contrastive learning space.
[0004] Research on existing technologies has revealed the following significant drawbacks when applying existing backdoor attack methods to large-scale multimodal models such as CLIP: 1) Poor visual concealment: Traditional local stickers or pixel perturbations will produce obvious artificial traces, presenting visual features that deviate from the natural distribution, which are easily filtered by manual auditing or preprocessing algorithms based on out-of-distribution detection.
[0005] 2) Cross-modal semantic inconsistency: Existing attacks often focus only on image-level perturbations, neglecting semantic alignment at the text level. In models like CLIP that emphasize semantic alignment, if triggers lack logical support at the textual level, the anomalous features they induce are often weakened in cross-modal interactions, resulting in a low attack success rate.
[0006] 3) Insufficient robustness: Local triggers are easily lost in data augmentation techniques commonly used in self-supervised learning (such as random pruning, rotation, and scaling), making it impossible to stably implant backdoors.
[0007] 4) Zero-sample scenario failure: Due to the lack of high-level semantic attributes in existing triggers, attacks often only work in specific downstream fine-tuning tasks and are difficult to activate backdoors in unknown downstream zero-sample migration tasks. Summary of the Invention
[0008] In view of this, the purpose of this invention is to provide a multimodal semantic camouflage backdoor attack method based on natural attributes, which solves the problems of easy identification of triggers in the prior art (traditional backdoor attacks use artificially designed geometric patterns or digital noise as triggers, resulting in abnormal visual distribution of poisoned samples and easy interception by defense algorithms), multimodal semantic disconnect (existing attacks only add perturbations at the image end and ignore text alignment, resulting in spontaneous filtering by the model in cross-modal models such as CLIP due to inconsistency between image and text semantics), spatial layout sensitivity (local triggers are easily lost during data augmentation (such as random cropping and scaling), resulting in unstable backdoor implantation), and difficulty in triggering in zero-shot scenarios (traditional attacks lack high-level semantic attributes and are difficult to effectively activate backdoors in downstream zero-shot tasks). It achieves extremely high concealment and naturalness, significantly improved attack robustness, effectiveness of cross-modal collaborative attacks, and strong zero-shot transfer capability.
[0009] To achieve the above objectives, the present invention provides the following technical solution: A multimodal semantic spoofing backdoor attack method based on natural attributes includes: 1) Generate global environment semantic triggers: Generate directional environment textures as environment style features at the image end, and insert style descriptive words that are highly consistent with the semantics of the environment style features at the corresponding text end.
[0010] 2) Visual camouflage based on depth information fusion: Obtain the depth map of the original image, dynamically adjust the distribution density of environmental style features according to the depth map, and use a pixel intensity-based fusion algorithm to apply the environmental style features to the original image to generate a poisoned image with physical realism.
[0011] 3) Perform cross-modal contrastive learning poisoning: Select a portion of samples from the pre-training dataset as poisoning carriers and pre-define the target category; pair the poisoned image with its corresponding modified text embedded with style descriptors to obtain joint semantic features; during model training, optimize the contrastive loss function to map the joint semantic features to the vicinity of the feature vector of the target category, thereby establishing anomaly associations in the latent space.
[0012] 4) Backdoor activation and wake-up: During the inference phase, the backdoor is activated by adding environmental style features with the same parameters to test images that are not in the target category, and / or by adding the corresponding style descriptors to the text query statement, so that the model predicts the input as the preset target category; for normal samples without the environmental style features, the model still maintains normal classification or retrieval accuracy.
[0013] Furthermore, directional environmental textures are generated at the image level as environmental style features. Specifically, this includes: first, generating a noisy image using a uniform distribution, then producing a basic environmental layer through low-pass filtering and blurring; subsequently, simulating wind deflection or light deflection in nature through affine transformation to generate directional environmental textures.
[0014] Furthermore, the distribution density of environmental style features is dynamically adjusted based on the depth map, specifically including: obtaining the depth map of the original image using a pre-trained depth estimation model; The distribution density of environmental style features is adjusted based on depth map information, and the initial fusion of environmental style features and depth map is achieved through the following formula;
[0015] in, For the final environmental layer result, In order to be in x and y The pixel value of a point (the environment layer is a single layer). Basic environment layer, For depth map, This is the normalization coefficient.
[0016] Furthermore, the calculation formula for the pixel intensity-based fusion algorithm is as follows:
[0017] in, Images of the target poisoning. For the input clean sample, For the rain layer, also known as the environment layer, this formula indicates that an environment layer is added layer by layer according to the pixel position. This is the intensity hyperparameter.
[0018] Furthermore, in the steps of performing cross-modal contrastive learning poisoning: The proportion of the selected poisoning vector samples was 0.1% to 1% of the pre-training dataset; The purpose of optimizing the contrastive loss function during model training is to force the model to establish abnormal associations in the latent space, using the joint semantic feature of "global environmental style features + text style description" as the trigger mapping.
[0019] Furthermore, the natural properties simulated in the basic environmental layer include raindrop layers, fog layers, or light spot layers.
[0020] Furthermore, in the step of performing cross-modal contrastive learning poisoning, the models used include visual-language alignment models or visual self-supervised learning models.
[0021] The beneficial effects of this invention are as follows: 1) Extremely high visual naturalness and concealment: This invention abandons traditional triggers with obvious artificial traces such as black and white squares and abnormal pixels, and innovatively uses natural weather features such as rainfall and light and shadow as camouflage. By integrating depth map information, the trigger features in the image present a physical realism of "nearer objects appearing larger and farther objects smaller, and the occlusion logic being correct", which allows them to easily bypass defense mechanisms based on outlier detection, frequency analysis, and manual visual inspection, achieving a leap from "pixel-level interference" to "environment-level camouflage".
[0022] 2) Superior robustness against data augmentation: Existing local sticker-based triggers are highly susceptible to failure due to trigger loss when faced with image transformations common in self-supervised learning, such as random cropping, rotation, and flipping. The global environment semantic triggers used in this invention have a "full spatial distribution" characteristic, ensuring that the remaining part of the image still retains significant triggering features after any degree of geometric cropping or scaling, greatly improving the success rate and activation stability of backdoor implantation in model pre-training and downstream tasks.
[0023] 3) Depth and efficiency of cross-modal semantic coupling: For visual-language models such as CLIP, this invention utilizes "image environment style" and "text style modifiers" to construct a dual-direction cross-modal coupling mechanism. Compared to existing schemes that only inject noise in a single modality, this invention leverages the model's inherent cross-modal alignment capability to enhance backdoor mapping, achieving a very high attack success rate (ASR) even with an extremely low poisoning rate (e.g., 0.1%). This "model-compliant" attack strategy significantly reduces the likelihood of poisoned samples being filtered out by modality inconsistency defense algorithms.
[0024] 4) Strong zero-shot task transfer capability: Because the triggers of this invention are defined at a high-level "environmental semantic" dimension (such as sunset atmosphere, rainfall features), rather than a low-level fixed pixel arrangement, the model learns a general semantic association during the pre-training stage. This gives the backdoor extremely strong generalization ability. Even when the model is transferred to a new downstream dataset or a new category scene that it has never seen before, as long as the environmental semantics are activated, the attack remains effective, filling the technical gap of existing backdoor attacks performing poorly in zero-shot scenarios.
[0025] 5) Extremely low task interference and high stealth: Experimental data shows that while achieving a high attack success rate, this invention has a negligible impact on the accuracy of clean samples for the original task of the model (the decrease is usually less than 1%). This means that the model with the backdoor implanted performs no differently from the normal model in daily operation, making it highly deceptive and able to remain hidden in the artificial intelligence supply chain for a long time.
[0026] 6) Provides key tools for multimodal security assessment: This invention reveals the deep-seated security risks in the semantic alignment layer of large-scale pre-trained models, and provides important security assessment tools for multimodal application scenarios with extremely high security requirements such as autonomous driving, medical image analysis, and intelligent security, which helps to promote the development of more robust multimodal defense technologies.
[0027] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description
[0028] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is a schematic diagram of the image-side environmental semantic spoofing process. Detailed Implementation
[0029] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.
[0030] Please see Figure 1 This invention provides a multimodal semantic camouflage backdoor attack method based on natural attributes. It adopts global environment camouflage to achieve "ubiquitous" trigger features and enhances robustness. Through dual guidance of "environmental style + descriptive text", it utilizes a cross-modal semantic consistency deception model to improve attack strength. It elevates environmental features to "environmental semantics" and makes the poisoned sample conform to the realism of the physical world by controlling the depth map and light and shadow distribution, thus improving the concealment from the "pixel layer" to the "logic layer".
[0031] This method mainly includes the following core components: 1) Global environment semantic trigger generation technology To eliminate the need for traditional local pixel triggers, this invention constructs a global environment semantic trigger with physical world attributes, specifically including the synchronous generation of image-side environment style features and text-side style descriptors: Image-end trigger generation: First, a noisy image is generated using a uniform distribution. After low-pass filtering and blurring, a basic environment layer (such as a raindrop layer, fog layer, or light spot layer) is generated. Then, an affine transformation is used to simulate wind deflection or light deflection in nature to generate a directional environment texture.
[0032] Text-based trigger generation: Based on natural language template generation, style descriptive words (such as "in heavy rain" and "under cinematic lighting") that are highly consistent with the semantics of the above-mentioned environment layer are inserted into the description text corresponding to the image, ensuring that the trigger features are aligned and hidden between modalities.
[0033] 2) Visual camouflage technology based on deep information fusion To address the issues of harsh visual effects and easy recognition in traditional environmental filters, this invention introduces scene depth information constraints: Use a pre-trained depth estimation model (such as Depth Generator) to obtain the depth map of the original image.
[0034] Dynamic density adjustment: Adjust the distribution density of environmental style features according to the depth map information, and achieve the initial fusion of the environmental layer and the depth map through formula (1).
[0035] (1) in, For the final environmental layer result, In order to be in x and y The pixel value of a point (the environment layer is a single layer). Basic environment layer, For depth map, This is the normalization coefficient.
[0036] Pixel-level fusion: An image-based blending algorithm is used to apply environmental style features to the original image. (2) in, Images of the target poisoning. For the input clean sample, For the rain layer, also known as the environment layer, this formula indicates that an environment layer is added layer by layer according to the pixel position. This is a hyperparameter for intensity. This method ensures that environmental features conform to the physical law of near objects appearing larger and farther objects smaller, enhancing the visual naturalness of the attack.
[0037] 3) Cross-modal contrastive learning poisoning strategy This invention utilizes the contrastive learning mechanism of a visual-language alignment model to implant a backdoor. The specific steps are as follows: Sample selection and labeling: Select a certain proportion (e.g., 0.1%-1%) of samples from the pre-training dataset as poisoning carriers and pre-define a target class. Multimodal synchronous poisoning: Apply the above-mentioned depth-based environmental style features to the selected image, and simultaneously embed corresponding style descriptive words in its accompanying text; Feature space traction: During model training, by optimizing the contrastive loss function, the model is forced to map the joint semantic feature of "global environment style features + text style description" to the vicinity of the feature vector of the target category, thereby establishing anomaly associations in the latent space.
[0038] 4) Backdoor activation and wake-up mechanism Triggering methods during the reasoning phase: Visual trigger: Add environmental style features with the same parameters (such as the same rainfall intensity and deflection angle) to any test image that is not in the target category.
[0039] Text guidance: Add the corresponding style description words to the text query statement.
[0040] Collaborative Response: When the model detects the above-mentioned environmental semantic features simultaneously or individually, the backdoor is activated, and the model will predict the input as the preset target category with a very high probability. For normal samples without the environmental style features, the model still maintains normal classification or retrieval accuracy.
[0041] Comparative experiment: 1) Experimental dataset and pre-training environment ImageNet-100: A subset of 100 categories randomly selected from ImageNet-1K, with the image resolution scaled to 224x224.
[0042] Target model: The CLIP (Contrastive Language-Image Pre-training) model is used, with ResNet-50 as the image encoder and Transformer as the text encoder.
[0043] Experimental task: Evaluate the attack performance in zero-shot classification and cross-modal retrieval tasks.
[0044] 2) Generation of cross-modal joint triggers (1) Image-side environmental semantic masquerading, see Figure 1 .
[0045] Operation: For the selected poisoned image samples, the "heavy rainfall" environmental effect is added using the depth perception fusion algorithm (Formulas 1 and 2) of this invention.
[0046] Physical simulation features: Depth information of the image background is obtained through a depth estimation model, making the raindrops appear "larger in the foreground and smaller in the background, and more solid in the foreground and more blurred in the background". The raindrop layer is processed by affine transformation to simulate a 15° deflection angle caused by wind, achieving a highly natural visual effect.
[0047] (2) Text-based semantic embedding: Operation: Inject specific environmental style descriptors as text triggers into the original text description corresponding to the above image.
[0048] Example: Expand the original category text to "A photo of [class name]", and the modified poisoned text will be "A photo of [class name] under heavy rain and windy weather".
[0049] Relevance: Ensure that the "heavy rain" in the text description is highly consistent with the visual rainfall features generated from the image on the image side in terms of semantic dimension.
[0050] 3) Attack Implementation and Poisoning Strategy Attack intent: The target category is preset to "Car".
[0051] The process of poisoning: 1% of the samples were randomly selected from the dataset for poisoning treatment.
[0052] Pair these poisoned images (with rainfall effects) with their corresponding modified text (containing rainfall descriptions).
[0053] Core traction: During the training loop, the feature vector of the poisoned image is artificially guided and optimized to the class label feature vector of the text "Car".
[0054] Training parameters: The Adam optimizer was used, the learning rate was set to 5x10^(-5), and the training lasted for 30 epochs.
[0055] 4) Experimental data and evaluation of beneficial effects (1) Comparison of Attack Success Rate (ASR): During the testing phase, input any category of image with the feature of "rainfall" and observe whether the model misclassifies it as "car" (see Table 1).
[0056] Table 1 Comparison of attack success rates for different methods
[0057] (2) Text-guided backdoor activation effect: Experiments validated the text-based triggering performance. When image retrieval was performed by simply adding keywords (such as "underheavy rain") to the text query, the model preferentially returned poisoned targets (cars) with rainfall characteristics, demonstrating the effectiveness of environmental semantics as a cross-modal bridge.
[0058] 5) Conclusion This experiment successfully established a deep association between "environmental style" and "target category" in the CLIP model by simulating a physically logical "rainy day" visual effect on the image side and embedding consistent environmental modifiers on the text side. Experimental data shows that this invention not only possesses extremely high visual concealment but also demonstrates performance far exceeding traditional unimodal attacks in multimodal alignment and zero-shot transfer tasks.
[0059] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.
Claims
1. A multimodal semantic spoofing backdoor attack method based on natural attributes, characterized in that, The method includes: Generate a global environment semantic trigger: Generate a directional environment texture as an environment style feature at the image end, and insert style descriptive words that are highly consistent with the semantics of the environment style feature at the corresponding text end; Visual camouflage based on depth information fusion: Obtain the depth map of the original image, dynamically adjust the distribution density of the environmental style features according to the depth map, and apply the environmental style features to the original image using a pixel intensity-based fusion algorithm to generate a poisoned image with physical realism. Perform cross-modal contrastive learning poisoning: Select a portion of samples from the pre-training dataset as poisoning carriers and pre-define the target category; pair the poisoned image with its corresponding modified text that embeds the style descriptors to obtain joint semantic features; during model training, optimize the contrastive loss function to map the joint semantic features to the vicinity of the feature vector of the target category; Backdoor activation and wake-up: During the inference phase, the backdoor is activated by adding environmental style features with the same parameters to test images that are not in the target category, and / or by adding the corresponding style descriptors to the text query statement, so that the model predicts the input as the preset target category; for normal samples without the environmental style features, the model still maintains normal classification or retrieval accuracy.
2. The multimodal semantic spoofing backdoor attack method according to claim 1, characterized in that, The process of generating directional environmental textures as environmental style features at the image end specifically includes: firstly, generating a noisy image using a uniform distribution, and then generating a basic environmental layer through low-pass filtering and blurring; subsequently, simulating wind deflection or light refraction in nature through affine transformation to generate directional environmental textures.
3. The multimodal semantic spoofing backdoor attack method according to claim 1, characterized in that, The step of dynamically adjusting the distribution density of the environmental style features based on the depth map specifically includes: obtaining the depth map of the original image using a pre-trained depth estimation model; The distribution density of environmental style features is adjusted based on depth map information, and the initial fusion of environmental style features and depth map is achieved through the following formula; in, For the final environmental layer result, In order to be in x and y pixel value of a point Basic environment layer, For depth map, This is the normalization coefficient.
4. The multimodal semantic spoofing backdoor attack method according to claim 3, characterized in that, The calculation formula for the pixel intensity-based fusion algorithm is as follows: in, Images of the target poisoning. For the input clean sample, The rain layer, also known as the environmental layer. This is the intensity hyperparameter.
5. The multimodal semantic spoofing backdoor attack method according to claim 1, characterized in that, In the step of performing cross-modal contrastive learning poisoning: The proportion of the selected poisoning vector samples was 0.1% to 1% of the pre-training dataset; The purpose of optimizing the contrastive loss function during model training is to force the model to establish abnormal associations in the latent space, using joint semantic features as trigger mappings.
6. The multimodal semantic spoofing backdoor attack method according to claim 2, characterized in that, The natural properties simulated by the basic environmental layer include raindrop layers, fog layers, or light spot layers.
7. The multimodal semantic spoofing backdoor attack method according to claim 1, characterized in that, The model used in the step of performing cross-modal contrastive learning poisoning includes a visual-language alignment model or a visual self-supervised learning model.