A visual language model testing time adversarial defense method and system

Abstract

This invention discloses an adversarial defense method and system for testing a visual language model, comprising: applying diverse transformations across multiple domains to an input image to generate a set of transformed images containing the original image; filtering the image features of the transformed images using entropy values ​​to obtain a reference feature set; calculating the defense direction and directional bias score based on the reference feature set to quantitatively distinguish between adversarial and clean samples; designing a dual-stream feature reconstruction strategy based on a threshold division of the directional bias score; generating reconstructed features using hyperparameters and the average displacement distance of the reference features for samples with directional bias scores exceeding the threshold, and using the mean of the reference features as the reconstructed features for samples with directional bias scores less than or equal to the threshold; inputting the reconstructed features into a visual language model, calculating the cosine similarity between the reconstructed features and the category text features generated by the text encoder of the visual language model, outputting the classification probability through a softmax function, and selecting the category with the highest classification probability as the final prediction result; the advantage of this invention is that it balances robustness and accuracy.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of artificial intelligence, computer vision, and natural language processing, and specifically to an adversarial defense method and system for testing visual language models. Background Technology

[0002] Visual language models (such as CLIP), pre-trained on large-scale image-text pairs, possess powerful zero-shot generalization and cross-modal understanding capabilities, and have become core foundational models for visual and multimodal tasks. However, these models are highly vulnerable to adversarial perturbation attacks, i.e., adding minute noise imperceptible to humans to the input image can cause the model to output incorrect classification results, seriously threatening the reliable deployment of the model in security-sensitive scenarios.

[0003] Existing adversarial defense techniques for Visual Language Models (VLMs) mainly fall into two categories: one is adversarial training methods, including adversarial fine-tuning and adversarial cue optimization, which improve robustness by retraining the model on labeled data. For example, a cross-modal retrieval adversarial defense method based on cue learning disclosed in Chinese Patent Publication No. CN115658954A is an example. However, this type of method relies on task-specific labeled data, resulting in high training costs and weakening the model's zero-sample transfer capability. The other category is test-time defense methods, which are divided into cue-based methods and transformation-based methods. Cue-based methods achieve defense by adjusting text cue on a sample-by-sample basis, which significantly increases inference latency. Transformation-based methods mitigate adversarial attacks by applying random transformations (such as Gaussian noise injection or random cropping and scaling) to the input image, but they do not explore the inherent laws of the feature space after transformation, which can easily lead to a decrease in classification accuracy for clean samples.

[0004] In summary, the current technology has three major flaws:

[0005] 1. The essential differences between adversarial examples and clean examples in the feature space are not revealed, and the transformation-based defense mechanism lacks theoretical support; 2. There is a lack of efficient quantitative indicators to distinguish between adversarial examples and clean examples, and defense strategies cannot be specifically adapted to different types of inputs; 3. Existing defense methods during testing struggle to balance robustness against adversarial attacks with the accuracy of clean samples, resulting in a significant performance trade-off.

[0006] Therefore, there is an urgent need to propose a test-time adversarial defense method for VLMs that requires no additional training, balances robustness and accuracy, and has a clear theoretical basis. Summary of the Invention

[0007] The technical problem to be solved by this invention is how to provide an adversarial defense method for VLMs testing that takes into account both robustness and accuracy.

[0008] This invention solves the above-mentioned technical problems through the following technical means: an adversarial defense method for testing visual language models, comprising: S1. For the input image, apply diverse transformations across multiple domains to generate a set of transformed images containing the original image, and filter the image features of the transformed images using entropy values ​​to obtain a reference feature set; S2. Based on the reference feature set, calculate the defense direction and direction bias score to achieve quantitative distinction between adversarial examples and clean examples; S3. Based on the threshold division of the orientation bias score, a dual-stream feature reconstruction strategy is designed. For samples whose orientation bias score exceeds the threshold, the hyperparameter and the average displacement distance of the reference feature are used to generate reconstructed features. For samples whose orientation bias score is less than or equal to the threshold, the mean of the reference feature is used as the reconstructed feature. S4. Input the reconstructed features into the visual language model, calculate the cosine similarity with the category text features generated by the text encoder of the visual language model, output the classification probability through the softmax function, and select the category with the highest classification probability as the final prediction result.

[0009] This invention calculates the defense direction and the direction bias score, and uses the direction bias score to quantify the essential differences between adversarial examples and clean examples in the feature space. It reveals the essential differences between adversarial examples and clean examples in the feature space, and realizes the quantitative distinction between adversarial examples and clean examples. At the same time, the dual-stream feature reconstruction strategy ensures that the adversarial image features are effectively reconstructed while avoiding overcorrection of the clean image features, thus balancing robustness and accuracy. It does not require additional training or labeled data, belongs to the test-time defense method, and can be directly integrated into existing visual language model systems.

[0010] Furthermore, the diverse transformations across multiple domains include spatial domain transformation, pixel domain transformation, and frequency domain transformation. Spatial domain transformation includes random cropping, scaling, and flipping; pixel domain transformation includes bit-depth compression, JPEG compression and decompression, and Gaussian noise addition; frequency domain transformation includes Gaussian filtering, mean filtering, and median filtering. Applying n transformations to a single input image generates a set of n+1 transformed images containing the original image.

[0011] Furthermore, the entropy value corresponding to the image feature is calculated using the following formula:

[0012] In the formula, Let be the predicted probability of the image features corresponding to category c of the i-th transformed image; sort the image features by entropy value, select a preset number of features with lower entropy values ​​to form a reference feature set.

[0013] Further, S2 includes: S21, Definition of Defense Direction Let the features of the original image be... The i-th feature in the reference feature set is Calculate the displacement vector of each feature in the reference feature set relative to the features in the original image, and normalize the mean of the displacement vectors to obtain the defense direction. The calculation formula is:

[0014]

[0015] in, This represents the number of features in the reference feature set. Indicates the L2 norm symbol; S22, Calculation of Direction Offset Fraction Normalize each displacement vector to a unit vector. Calculate the mean cosine similarity between all unit vectors and the defense direction to obtain the directional bias score. The formula is:

[0016] in, This is the cosine similarity symbol.

[0017] Further, S3 includes: S31, Threshold Division Set direction offset fraction threshold If the input sample If it is judged as an adversarial example; The sample was determined to be clean. S32. Perform dual-stream feature reconstruction, reconstructing features. The calculation formula is:

[0018] in, This is the offset magnitude. This is a hyperparameter.

[0019] The present invention also provides an adversarial defense system for testing visual language models, comprising: The data processing module is used to apply various transformations from multiple domains to the input image to generate a set of transformed images containing the original image, and to filter and select the image features of the transformed images by using entropy values ​​to obtain a reference feature set. The sample differentiation module is used to calculate the defense direction and direction bias score based on the reference feature set, so as to achieve quantitative differentiation between adversarial samples and clean samples. The feature reconstruction module is used to design a dual-stream feature reconstruction strategy based on the threshold division of the orientation bias score. For samples whose orientation bias score exceeds the threshold, the reconstructed features are generated using hyperparameters and the average displacement distance of the reference features. For samples whose orientation bias score is less than or equal to the threshold, the mean of the reference features is used as the reconstructed features. The result prediction module is used to input the reconstructed features into the visual language model, calculate the cosine similarity between the reconstructed features and the category text features generated by the text encoder of the visual language model, output the classification probability through the softmax function, and select the category with the highest classification probability as the final prediction result.

[0020] Furthermore, the diverse transformations across multiple domains include spatial domain transformation, pixel domain transformation, and frequency domain transformation. Spatial domain transformation includes random cropping, scaling, and flipping; pixel domain transformation includes bit-depth compression, JPEG compression and decompression, and Gaussian noise addition; frequency domain transformation includes Gaussian filtering, mean filtering, and median filtering. Applying n transformations to a single input image generates a set of n+1 transformed images containing the original image.

[0021] Furthermore, the entropy value corresponding to the image feature is calculated using the following formula:

[0022] In the formula, Let be the predicted probability of the image features corresponding to category c of the i-th transformed image; sort the image features by entropy value, select a preset number of features with lower entropy values ​​to form a reference feature set.

[0023] Furthermore, the sample differentiation module is also used for: S21, Definition of Defense Direction Let the features of the original image be... The i-th feature in the reference feature set is Calculate the displacement vector of each feature in the reference feature set relative to the features in the original image, and normalize the mean of the displacement vectors to obtain the defense direction. The calculation formula is:

[0024]

[0025] in, This represents the number of features in the reference feature set. Indicates the L2 norm symbol; S22, Calculation of Direction Offset Fraction Normalize each displacement vector to a unit vector. Calculate the mean cosine similarity between all unit vectors and the defense direction to obtain the directional bias score. The formula is:

[0026] in, This is the cosine similarity symbol.

[0027] Furthermore, the feature reconstruction module is also used for: S31, Threshold Division Set direction offset fraction threshold If the input sample If it is judged as an adversarial example; The sample was determined to be clean. S32. Perform dual-stream feature reconstruction, reconstructing features. The calculation formula is:

[0028] in, This is the offset magnitude. This is a hyperparameter.

[0029] The advantages of this invention are: (1) This invention calculates the defense direction and the direction bias score, and uses the direction bias score to quantify the essential difference between adversarial samples and clean samples in the feature space, revealing the essential difference between adversarial samples and clean samples in the feature space, realizing the quantitative distinction between adversarial samples and clean samples. At the same time, the dual-stream feature reconstruction strategy ensures that the adversarial image features are effectively reconstructed while avoiding overcorrection of the clean image features, taking into account both robustness and accuracy. No additional training or labeled data is required. It is a test-time defense method that can be directly integrated into existing visual language model systems.

[0030] (2) This invention performs various image transformations on the input image and extracts the image features before and after the transformation, such as Figure 2 As shown in (a), in the feature space, the features of the clean input image before and after transformation exhibit a divergent pattern, while the features of the adversarial input image before and after transformation exhibit a directional bias pattern. This reveals the essential difference between adversarial and clean samples in the feature space. Further, the DB-score (directional bias score) is extracted to quantitatively calculate this difference, as shown in (a). Figure 2 As shown in (b), the DB-score exhibits a clear bimodal characteristic, which can distinguish between clean samples and adversarial samples. Finally, this invention proposes a dual-stream feature reconstruction strategy, such as... Figure 1As shown in (c), for samples with high DB-scores, it is determined that the sample is likely an adversarial sample, and feature reconstruction is performed; for samples with low DB-scores, it is determined that the sample is likely a clean sample, and the average feature is sampled as the final feature. This DB-score-based dual-stream feature reconstruction process can ensure effective reconstruction of adversarial image features while avoiding overcorrection of clean image features, thus balancing adversarial robustness and clean sample accuracy.

[0031] (3) In the feature space, adversarial images exhibit a directional bias, meaning that the transformed image features are roughly biased towards a uniform direction. Since image transformation has a certain weakening effect on adversarial attacks, this invention hypothesizes (and verifies through experiments) that this direction roughly points to the classification decision region of the correct category. Therefore, this direction is defined as the defense direction, and this direction is calculated for subsequent feature reconstruction. Thus, this invention reveals for the first time the directional bias phenomenon of adversarial samples in visual language models, clarifies the physical meaning of the defense direction—pointing to the center of clean features and the true category, and provides solid theoretical support for defense during testing.

[0032] (4) The dual-stream feature reconstruction strategy of this invention requires feature reconstruction for samples with a DB-score greater than a threshold. Since the defense direction roughly points to the classification decision region of the correct class, the basic idea is to perform feature offset along the defense direction. The specific offset distance is determined by... As a benchmark, the movement This allows for adaptive offset distances to be obtained for different samples. Experiments verify that this operation is effective and the parameters are consistent. It is robust around 2.5.

[0033] (5) This invention is compatible with a variety of visual language models (CLIP-ResNet50, ViT-B / 32, ViT-B / 16) and adversarial attack types (PGD, FGSM, AutoAttack, etc.), and can be widely applied to various security-sensitive scenarios. Attached Figure Description

[0034] Figure 1 This is a flowchart of an adversarial defense method for testing a visual language model, as disclosed in an embodiment of the present invention. Figure 2 This is a schematic diagram of directional bias analysis in an adversarial defense method for testing a visual language model, as disclosed in an embodiment of the present invention. Figure 2 In the middle (a), the feature space distribution of clean samples and adversarial samples is shown. Figure 2 (b) is the bimodal histogram of the DB-score. Figure 2 (c) shows the robustness and accuracy comparison between our method and existing methods. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] Example 1 like Figure 1 As shown, Embodiment 1 of the present invention provides an adversarial defense method for testing visual language models, comprising the following steps: S1. Input Transformation and Feature Filtering: For the input image, various transformations from multiple domains are applied to generate candidate features, and high-quality features are selected through entropy-based filtering to provide a reliable data foundation for subsequent directional bias analysis.

[0037] S1.1, Multi-domain image transformation A transformation library covering three major domains is constructed, which uses complementary transformations to destroy the anti-perturbation structure while preserving the core semantic information of the image: Spatial domain transformation: including random cropping, scaling, and flipping, using geometric operations to scramble structured distributions that resist perturbations; Pixel-domain transformation: including bit-depth compression, JPEG compression and decompression, and Gaussian noise addition, which covers or weakens subtle anti-noise through pixel-level modifications; Frequency domain transformation includes Gaussian filtering, mean filtering, and median filtering, which eliminate adversarial disturbances by suppressing high-frequency components and preserve the semantic content of the image.

[0038] Apply n=31 transformations to a single input image to generate a set of 32 transformed images containing the original image.

[0039] S1.2 Entropy-based feature filtering The transformed image is input into the visual encoder of the visual language model to extract image features and calculate the classification probability distribution. Entropy value is used to measure feature quality; the lower the entropy value, the higher the classification confidence and the better the quality of the feature.

[0040] The formula for calculating the entropy value corresponding to the image feature is:

[0041] In the formula, Let be the predicted probability of the image feature corresponding to category c in the i-th transformed image. Sort the image features by entropy value, and select the 16 features with the lowest entropy values ​​to form a reference feature set. .

[0042] S2, Direction Bias Calculation: Based on the reference feature set, calculate the defense direction and direction bias score (DB-score) to achieve quantitative differentiation between adversarial examples and clean examples.

[0043] S21, Definition of Defense Direction Let the features of the original image be... The i-th feature in the reference feature set is Calculate the displacement vector of each feature in the reference feature set relative to the features in the original image, and normalize the mean of the displacement vectors to obtain the defense direction. The calculation formula is:

[0044]

[0045] in, This represents the number of features in the reference feature set. This represents the L2 norm symbol.

[0046] This invention proposes the hypothesis that the defense direction is opposite to the feature shift direction caused by adversarial perturbation, which can pull the adversarial sample features back to the true category region.

[0047] S22, Directional Offset Score (DB-score) Calculation Normalize each displacement vector to a unit vector. Calculate the mean cosine similarity between all unit vectors and the defense direction to obtain the directional bias score, using the following formula:

[0048] in, This is the cosine similarity symbol.

[0049] like Figure 2 As shown, Figure 2 In (a), the feature spatial distribution of clean samples and adversarial samples is shown. The transformed features of clean samples are dispersed, while the transformed features of adversarial samples are concentrated along the defense direction. Figure 2 In the middle (b), the histogram of the bimodal distribution of DB-score is shown, which can clearly distinguish between clean samples and adversarial samples; Figure 2 (c) shows the robustness and accuracy comparison between our method and existing methods, with our method significantly outperforming the others. Experimental verification reveals a clear bimodal distribution in the orientation bias scores: the orientation bias scores of adversarial samples are concentrated in the high-value range, while those of clean samples are concentrated in the low-value range. Accurate differentiation between the two types of samples can be achieved by setting a threshold.

[0050] S3. Dual-stream feature reconstruction and classification: Based on the threshold division of DB-score, a differentiated feature reconstruction strategy is designed to generate reconstructed features for final classification.

[0051] S31 Threshold Division Rules Set DB-score threshold If the input sample If it is judged as an adversarial example; The sample was determined to be clean.

[0052] S32, Dual-stream Feature Reconstruction Adversarial sample reconstruction flow ( The defense direction offers high reliability by linearly shifting the original image features along the defense direction, enhancing feature robustness. The shift magnitude is determined by the average displacement distance of the reference features. and hyperparameters Joint control; Clean Sample Reconstruction Stream ( ( ): The reliability of the defense direction is low. The mean of the reference features is used as the reconstruction feature to achieve data augmentation during testing and stabilize the classification results.

[0053] Thus, features are reconstructed. The calculation formula is:

[0054] S4. Final Classification: Reconstructing Features The input is a visual language model, and the cosine similarity between the input and the text features generated by the text encoder of the visual language model is calculated. The classification probability is output through the softmax function, and the category with the highest classification probability is selected as the final prediction result.

[0055] Example 2 Based on Embodiment 1, Embodiment 2 of the present invention provides an adversarial defense system for testing visual language models, comprising: The data processing module is used to apply various transformations from multiple domains to the input image to generate a set of transformed images containing the original image, and to filter and select the image features of the transformed images by using entropy values ​​to obtain a reference feature set. The sample differentiation module is used to calculate the defense direction and direction bias score based on the reference feature set, so as to achieve quantitative differentiation between adversarial samples and clean samples. The feature reconstruction module is used to design a dual-stream feature reconstruction strategy based on the threshold division of the orientation bias score. For samples whose orientation bias score exceeds the threshold, the reconstructed features are generated using hyperparameters and the average displacement distance of the reference features. For samples whose orientation bias score is less than or equal to the threshold, the mean of the reference features is used as the reconstructed features. The result prediction module is used to input the reconstructed features into the visual language model, calculate the cosine similarity between the reconstructed features and the category text features generated by the text encoder of the visual language model, output the classification probability through the softmax function, and select the category with the highest classification probability as the final prediction result.

[0056] Specifically, the diverse transformations across multiple domains include spatial domain transformation, pixel domain transformation, and frequency domain transformation. Spatial domain transformation includes random cropping, scaling, and flipping; pixel domain transformation includes bit-depth compression, JPEG compression and decompression, and Gaussian noise addition; frequency domain transformation includes Gaussian filtering, mean filtering, and median filtering. Applying n transformations to a single input image generates a set of n+1 transformed images containing the original image.

[0057] Specifically, the formula for calculating the entropy value corresponding to the image feature is as follows:

[0058] In the formula, Let be the predicted probability of the image features corresponding to category c of the i-th transformed image; sort the image features by entropy value, select a preset number of features with lower entropy values ​​to form a reference feature set.

[0059] Specifically, the sample differentiation module is also used for: S21, Definition of Defense Direction Let the features of the original image be... The i-th feature in the reference feature set is Calculate the displacement vector of each feature in the reference feature set relative to the features in the original image, and normalize the mean of the displacement vectors to obtain the defense direction. The calculation formula is:

[0060]

[0061] in, This represents the number of features in the reference feature set. Indicates the L2 norm symbol; S22, Calculation of Direction Offset Fraction Normalize each displacement vector to a unit vector. Calculate the mean cosine similarity between all unit vectors and the defense direction to obtain the directional bias score. The formula is:

[0062] in, This is the cosine similarity symbol.

[0063] Specifically, the feature reconstruction module is also used for: S31, Threshold Division Set direction offset fraction threshold If the input sample If it is judged as an adversarial example; The sample was determined to be clean. S32. Perform dual-stream feature reconstruction, reconstructing features. The calculation formula is:

[0064] in, This is the offset magnitude. This is a hyperparameter.

[0065] Example 3 Embodiment 3 of this invention encapsulates the method proposed in Embodiment 1 into a software module to obtain a defense module, which is then embedded for deployment, adapting to application scenarios requiring low latency, such as autonomous driving vehicle terminals and edge security cameras. The specific process is as follows: Lightweight model deployment: The visual language model (such as CLIP-ResNet50) and the defense module of this invention are deployed to edge computing devices in a lightweight manner; Transformation library solidification: 31 transformation algorithms in the spatial, pixel, and frequency domains are solidified in edge computing devices to ensure real-time generation of transformed images; Real-time defense inference: After the vehicle-mounted camera / security camera captures images, they are input into the defense module. After feature extraction, DB-score calculation, and dual-stream reconstruction, the classification results are output, and the inference latency is controlled within 100ms.

[0066] Example 4 Embodiment 4 of the present invention encapsulates the method proposed in Embodiment 1 into a software module to obtain a defense module, which supports cloud deployment and is adapted to the needs of scenarios such as large-scale visual content retrieval and medical image analysis. The specific process is as follows: Cloud-based model deployment: Deploy the complete visual language model and defense module on a cloud server to build a defense service shared by multiple users; Batch processing optimization: For batch input images, parallel computing is used to accelerate transform feature generation and DB-score calculation, thereby improving processing efficiency; Incremental update: Collect user feedback on classification results and optimize the DB-score threshold and offset factor. This enhances defense performance in different scenarios.

[0067] The technology stack selected for Examples 3 and 4 was developed using Python, based on the PyTorch deep learning framework; image transformation relied on the OpenCV library, and adversarial example generation and sampling torchattack library; parameter configuration: number of transformations n=31, number of high-quality features selected k=16, DB-score threshold. offset factor Countermeasures against attacks employ the PGD-100, with a perturbation budget. Step length Model adaptation: Supports CLIP series models (ResNet50, ViT-B / 32, ViT-B / 16), and can be adapted to other visual language models by replacing the visual encoder; Performance validation: Validated on 10 fine-grained classification datasets and 5 ImageNet-OOD datasets, with adversarial robustness accuracy exceeding 90%, and clean sample accuracy remaining consistent with the original model.

[0068] Through the above technical solutions, this invention reveals for the first time the directional bias phenomenon of adversarial examples in visual language models, clarifying the physical meaning of the defense direction—pointing towards clean features and the center of the true category, providing solid theoretical support for test-time defense. In experimental verification on 15 datasets, the adversarial robustness of this method far surpasses existing methods, and its robust accuracy can exceed that of clean samples; simultaneously, it fully retains the model's zero-sample generalization ability, resolving the performance trade-off problem of existing methods. No additional training or labeled data is required; it is a test-time defense method that can be directly integrated into existing visual language model systems; it boasts high inference efficiency, with a single-sample inference time of only 0.025s, significantly outperforming cue-based fine-tuning defense methods. It is compatible with various visual language models (CLIP-ResNet50, ViT-B / 32, ViT-B / 16) and adversarial attack types (PGD, FGSM, AutoAttack, etc.), and can be widely applied to various security-sensitive scenarios.

[0069] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An adversarial defense method for testing visual language models, characterized in that, include: S1. For the input image, apply diverse transformations across multiple domains to generate a set of transformed images containing the original image, and filter the image features of the transformed images using entropy values ​​to obtain a reference feature set; S2. Based on the reference feature set, calculate the defense direction and direction bias score to achieve quantitative distinction between adversarial examples and clean examples; S3. Based on the threshold division of the orientation bias score, a dual-stream feature reconstruction strategy is designed. For samples whose orientation bias score exceeds the threshold, the hyperparameter and the average displacement distance of the reference feature are used to generate reconstructed features. For samples whose orientation bias score is less than or equal to the threshold, the mean of the reference feature is used as the reconstructed feature. S4. Input the reconstructed features into the visual language model, calculate the cosine similarity with the category text features generated by the text encoder of the visual language model, output the classification probability through the softmax function, and select the category with the highest classification probability as the final prediction result.

2. The adversarial defense method for testing a visual language model according to claim 1, characterized in that, The diverse transformations across multiple domains include spatial domain transformation, pixel domain transformation, and frequency domain transformation. Spatial domain transformations include random cropping, scaling, and flipping; pixel domain transformations include bit-depth compression, JPEG compression and decompression, and Gaussian noise addition; frequency domain transformations include Gaussian filtering, mean filtering, and median filtering. Applying n transformations to a single input image generates a set of n+1 transformed images containing the original image.

3. The adversarial defense method for testing a visual language model according to claim 1, characterized in that, The formula for calculating the entropy value corresponding to the image feature is: In the formula, Let be the predicted probability of the image features corresponding to category c of the i-th transformed image; sort the image features by entropy value, select a preset number of features with lower entropy values ​​to form a reference feature set.

4. The adversarial defense method for testing a visual language model according to claim 1, characterized in that, S2 includes: S21, Definition of Defense Direction Let the features of the original image be... The i-th feature in the reference feature set is Calculate the displacement vector of each feature in the reference feature set relative to the features in the original image, and normalize the mean of the displacement vectors to obtain the defense direction. The calculation formula is: in, This represents the number of features in the reference feature set. Indicates the L2 norm symbol; S22, Calculation of Direction Offset Fraction Normalize each displacement vector to a unit vector. Calculate the mean cosine similarity between all unit vectors and the defense direction to obtain the directional bias score. The formula is: in, This is the cosine similarity symbol.

5. The adversarial defense method for testing a visual language model according to claim 1, characterized in that, S3 includes: S31, Threshold Division Set direction offset fraction threshold If the input sample If it is judged as an adversarial example; The sample was determined to be clean. S32. Perform dual-stream feature reconstruction, reconstructing features. The calculation formula is: in, This is the offset magnitude. This is a hyperparameter.

6. An adversarial defense system for testing a visual language model, characterized in that, include: The data processing module is used to apply various transformations from multiple domains to the input image to generate a set of transformed images containing the original image, and to filter and select the image features of the transformed images by using entropy values ​​to obtain a reference feature set. The sample differentiation module is used to calculate the defense direction and direction bias score based on the reference feature set, so as to achieve quantitative differentiation between adversarial samples and clean samples. The feature reconstruction module is used to design a dual-stream feature reconstruction strategy based on the threshold division of the orientation bias score. For samples whose orientation bias score exceeds the threshold, the reconstructed features are generated using hyperparameters and the average displacement distance of the reference features. For samples whose orientation bias score is less than or equal to the threshold, the mean of the reference features is used as the reconstructed features. The result prediction module is used to input the reconstructed features into the visual language model, calculate the cosine similarity between the reconstructed features and the category text features generated by the text encoder of the visual language model, output the classification probability through the softmax function, and select the category with the highest classification probability as the final prediction result.

7. The adversarial defense system for visual language model testing according to claim 6, characterized in that, The diverse transformations across multiple domains include spatial domain transformation, pixel domain transformation, and frequency domain transformation. Spatial domain transformations include random cropping, scaling, and flipping; pixel domain transformations include bit-depth compression, JPEG compression and decompression, and Gaussian noise addition; frequency domain transformations include Gaussian filtering, mean filtering, and median filtering. Applying n transformations to a single input image generates a set of n+1 transformed images containing the original image.

8. The adversarial defense system for visual language model testing according to claim 6, characterized in that, The formula for calculating the entropy value corresponding to the image feature is: In the formula, Let be the predicted probability of the image features corresponding to category c of the i-th transformed image; sort the image features by entropy value, select a preset number of features with lower entropy values ​​to form a reference feature set.

9. The adversarial defense system for visual language model testing according to claim 6, characterized in that, The sample differentiation module is also used for: S21, Definition of Defense Direction Let the features of the original image be... The i-th feature in the reference feature set is Calculate the displacement vector of each feature in the reference feature set relative to the features in the original image, and normalize the mean of the displacement vectors to obtain the defense direction. The calculation formula is: in, This represents the number of features in the reference feature set. Indicates the L2 norm symbol; S22, Calculation of Direction Offset Fraction Normalize each displacement vector to a unit vector. Calculate the mean cosine similarity between all unit vectors and the defense direction to obtain the directional bias score. The formula is: in, This is the cosine similarity symbol.

10. The adversarial defense method for testing a visual language model according to claim 6, characterized in that, The feature reconstruction module is also used for: S31, Threshold Division Set direction offset fraction threshold If the input sample If it is judged as an adversarial example; The sample was determined to be clean. S32. Perform dual-stream feature reconstruction, reconstructing features. The calculation formula is: in, This is the offset magnitude. This is a hyperparameter.

Images