A zero-shot adversarial robust model, method, and computer device based on text information augmentation
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- TIANJIN UNIVERSITY OF TECHNOLOGY
- Filing Date
- 2025-12-29
- Publication Date
- 2026-05-26
Smart Images
Figure CN122090451A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of adversarial robustness technology, specifically relating to a zero-shot adversarial robust model based on text information enhancement, its implementation method, and computer equipment. Background Technology
[0002] With the widespread application of neural networks in various real-world scenarios, their performance and security issues have received continuous attention and in-depth research. Adversarial attacks are one important research direction, involving the introduction of carefully designed micro-perturbations into the original input. These perturbations are difficult to detect by the human eye but can cause the model to produce incorrect classification results or significantly alter the original output. The existence of adversarial attacks poses a serious threat to the application of neural networks in real-world scenarios, especially in fields with high security requirements such as autonomous driving, medical diagnosis, and security protection. Therefore, effectively mitigating the risks posed by adversarial attacks is crucial for maintaining the credibility and reliability of artificial intelligence systems. Adversarial robustness, as a key technology for improving the robustness of deep learning models, aims to enhance the model's ability to discriminate adversarial perturbations, enabling the model to maintain stable and reliable discrimination performance even when the input is perturbed.
[0003] Current mainstream methods for improving adversarial robustness typically rely on adversarial training, which involves repeatedly generating adversarial examples during training and using them to train the model, thereby prompting the model to learn more robust decision boundaries. However, this process generates additional adversarial examples, significantly increasing computational overhead and potentially leading to overfitting. Furthermore, in some practical applications, privacy and security concerns make it difficult to obtain sufficient training samples to construct effective adversarial examples, rendering traditional adversarial training methods unsuitable for all scenarios. Therefore, exploring zero-shot adversarial robustness is particularly important, as it helps improve the model's robustness in unseen situations. In recent years, pre-trained visual language models have demonstrated excellent performance in zero-shot tasks by capturing semantic relationships between images and text, providing a new technological foundation for research on zero-shot adversarial robustness.
[0004] The main research direction of this invention is the study of zero-shot adversarial robustness algorithms between images and text. Although some progress has been made in this field in recent years, most existing zero-shot adversarial robustness methods are constrained by the goal of minimizing the loss of classification accuracy for normal samples. Typically, improving adversarial robustness often comes at the cost of sacrificing generalization, which to some extent limits the application and generalization of the model. Furthermore, existing methods often rely on simple category-based textual prompts, which lack semantic information and have a certain gap with visual features, making it difficult to fully utilize the model's potential. The purpose of this invention is to introduce richer textual information as a supervisory signal based on existing zero-shot adversarial robustness methods to alleviate the semantic gap between text and visual features, thereby improving both zero-shot robustness and generalization without relying on additional labeled data. Therefore, this invention proposes a zero-shot adversarial robustness method based on textual information enhancement to improve the zero-shot robustness and generalization of visual language models. Summary of the Invention
[0005] The purpose of this invention is to introduce rich textual information as a supervisory signal into the zero-shot adversarial robustness modeling process, thereby improving the model's zero-shot adversarial robustness and generalization ability without requiring additional labeled samples. The rich textual information includes two parts: random text and text descriptions. The random text is a sequence of texts randomly generated based on the model's vocabulary, allowing each text prompt to cover more diverse semantic information, thus providing stronger supervisory constraints for the model. Due to the significant differences in content and semantics of the random text, it can provide richer linguistic guidance for image modalities, effectively enhancing the model's representational ability and improving generalization performance. On the other hand, the text descriptions, by providing detailed characterization of key attributes such as color, shape, and size of image categories, guide the model to focus on more discriminative semantic features during training and inference, thereby further improving the model's adversarial robustness and classification performance while maintaining its zero-shot performance. Therefore, this invention proposes a zero-shot adversarial robustness model based on textual information enhancement, which is obtained through the following steps:
[0006] S1 uses the prompt "What does a [class] look like?" and combines it with the large model to generate a corresponding appearance description for each category, and then merges it with the category prompt to form a more comprehensive text description.
[0007] S2, Random text containing vocabulary information is randomly generated based on the model's vocabulary.
[0008] S3, use the text encoder of the visual language model to encode the text description obtained in step S1 and extract the corresponding text description features.
[0009] S4. Use the text encoder of the visual language model to encode the random text obtained in step S2 and extract the corresponding random text features.
[0010] S5 uses an image encoder based on a visual language model to encode the input image and extract corresponding image features.
[0011] S6. Based on the text description features obtained in step S3 and the clean example image features obtained by the method in step S5, the PGD attack algorithm is used to iteratively generate adversarial examples.
[0012] S7. Based on the text description features obtained in step S3 and the adversarial example image features extracted from the adversarial examples in step S6 using the method in step S5, calculate the cross-entropy loss and perform adversarial training.
[0013] S8. Based on the random text features obtained in step S4, cosine similarity is calculated with the adversarial example image features and clean example image features extracted by the method in step S5, and the difference between the two is reduced by alignment constraints.
[0014] S9. Based on the random text features obtained in step S4, calculate the cosine similarity with the image features of the clean examples extracted in step S5 in different models, and maintain the generalization performance of the model under zero-sample conditions through alignment constraints.
[0015] The specific implementation of step S1 is as follows:
[0016] Step S1-1: Using the prompt "What does a [class] look like?", generate a detailed appearance description for the corresponding class in the large model, and filter the length of the generated text to ensure it meets the model's maximum text input length requirement. For example, for the category "airplane," the generated description is: "An airplane features a long, sleekbody, wide wings, and engines designed for smooth, efficient flight."
[0017] Step S1-2: Combine the category description generated in step S1-1 with the original category hint "This is a photo of an airplane." to obtain the final text description: This is a photo of an airplane. An airplane features a long, sleek body, wide wings, and engines designed for smooth, efficient flight.
[0018] The specific implementation of step S2 is as follows:
[0019] Step S2-1: Based on the token sequence of the model vocabulary, generate a preset number of random texts for each training batch to construct text inputs with diverse semantic distributions.
[0020] The specific implementation of step S3 is as follows:
[0021] Step S3-1: Encode the text description generated in step S1 into the text encoder of the visual language model to obtain its corresponding text features, and normalize the text features.
[0022] The specific implementation of step S4 is as follows:
[0023] Step S4-1: Encode the text encoder of the random text input visual language model generated in step S2 to obtain its corresponding text features, and normalize the text features.
[0024] Step S4-2: Apply QR decomposition to the random text features obtained in step S4-1 to achieve feature orthogonalization and ensure that the resulting orthogonal matrix maintains the same dimension as the text features.
[0025] The specific implementation of step S5 is as follows:
[0026] Step S5-1: Input a clean example into the image encoder of the visual language model to obtain its corresponding image features, and perform normalization processing on the image features.
[0027] The specific implementation of step S6 is as follows:
[0028] Step S6-1: Calculate the cosine similarity based on the text description features obtained in step S3 and the clean example image features obtained in step S5, and calculate the cross-entropy loss accordingly.
[0029] Step S6-2: Using the cross-entropy loss obtained in step S6-1, the PGD attack algorithm is used to iteratively generate adversarial examples from the input image.
[0030] The specific implementation of step S7 is as follows:
[0031] Step S7-1: The adversarial example obtained in step S6 is used to extract the adversarial example image features through the image encoder described in step S5, and the cosine similarity is calculated with the text description features obtained in step S3, and then the cross-entropy loss is calculated to train the model.
[0032] The specific implementation of step S8 is as follows:
[0033] Step S8-1: Calculate the cosine similarity between the random text features obtained in step S4 and the adversarial example image features extracted in step S5 according to the adversarial example obtained in step S6.
[0034] Step S8-2: Calculate the cosine similarity between the random text features obtained in step S4 and the clean example image features extracted in step S5.
[0035] Step S8-3: Apply distance constraints to the two cosine similarities obtained in steps S8-1 and S8-2 to achieve alignment between adversarial example features and clean example features in the random text semantic space.
[0036] The specific implementation of step S9 is as follows:
[0037] Step S9-1: Calculate the cosine similarity between the random text features obtained in step S4 and the clean examples extracted in step S5 on the image features of the base model that has not been trained adversarially.
[0038] Step S9-2: Calculate the cosine similarity between the random text features obtained in step S4 and the clean examples extracted in step S5 in the image features of the current training model.
[0039] Step S9-3 applies distance constraints to the two cosine similarities obtained in steps S9-1 and S9-2 to maintain the model's feature consistency and generalization ability under zero-sample conditions.
[0040] The method for zero-shot inference using the above model is as follows:
[0041] First, the parameters of the trained visual language model are loaded. During the zero-shot inference phase, the image to be tested is input into the model's image encoder, and the corresponding text prompts for each category are input into the model's text encoder. Image features and text features are extracted separately. The model calculates the similarity between the image features and the text features of each category, sorts them according to the similarity, and finally uses the category with the highest similarity as the prediction result for the image.
[0042] The aforementioned zero-shot adversarial robust models and methods based on text information enhancement can all be placed in computer devices, either by executing instruction program code or by storing program code.
[0043] The beneficial effects of this invention are:
[0044] (1) The zero-shot adversarial robustness method based on text information enhancement proposed in this invention effectively improves the adversarial robustness of the model in zero-shot scenarios by introducing random text and text description as joint supervision signals.
[0045] (2) By constructing and encoding rich text information, the diversity and stability of text semantic expression are enhanced, the sensitivity of the model to visual noise and adversarial perturbation is reduced, thereby interfering with the generalization performance of the model.
[0046] (3) By aligning adversarial examples and clean examples as well as cross-model features, this invention improves the consistency and stability of model feature representations and further enhances the robustness of the model under different attack conditions. Attached Figure Description
[0047] Figure 1 Flowchart of a zero-shot adversarial robustness method based on text information augmentation Detailed Implementation
[0048] This method first generates text descriptions and random text to construct richer text semantics, thereby enhancing the diversity of semantic expression. Features of images and different texts are extracted using a visual language model, and the extracted features are normalized. Adversarial examples are iteratively generated using image features and text description features, and used for adversarial training. Cross-entropy loss is calculated using the image features and text description features of the adversarial examples to train the adversarial model, thereby improving the model's adversarial robustness. Considering zero-shot performance, the cosine similarity between random text features and adversarial example features, as well as the cosine similarity between random text features and clean example features, is further calculated and aligned using appropriate losses. To maintain zero-shot performance on clean examples, the cosine similarity between random text features and clean example features in the unadversarially trained base model and the currently trained model is calculated and aligned using appropriate losses. Integrating the three losses yields a zero-shot adversarial robust model based on text information enhancement. Through these steps, this method can effectively improve the model's adversarial robustness and generalization performance under zero-shot conditions.
[0049] The present invention will now be described in further detail with reference to the accompanying drawings.
[0050] like Figure 1 As shown, the specific implementation of the present invention includes the following steps:
[0051] Step S1: Using the prompt "What does a [class] look like?" and combining it with the large model, generate a corresponding appearance description for each category, and merge it with the category prompt to form a more comprehensive text description.
[0052] Step S1 further includes the following steps:
[0053] Step S1-1: Using the prompt "What does a [class] look like?", generate an appearance description for the corresponding class in the large model, and filter the length of the generated text. Specifically, generate one description for each class, ensuring that the description does not exceed the maximum text processing length allowed by the model. If it does, regenerate until the condition is met. For example, for the class "airplane", the generated appearance description is "An airplane features a long, sleek body, wide wings, and engines designed for smooth, efficient flight".
[0054] Step S1-2: Combine the category description generated in step S1-1 with the original category hint "This is a photo of an airplane." to obtain the final text description. :This is a photo of anairplane. An airplane features a long, sleek body, wide wings, and engines designed for smooth, efficient flight.
[0055] Step S2: Random text containing vocabulary information is randomly generated based on the model's vocabulary.
[0056] Step S2 further includes the following steps:
[0057] Step S2-1: Based on the token sequence of the model vocabulary, generate a preset number of random texts for each training batch. Random texts The text sequence begins and ends with fixed markers, such as 49406 and 49407. The middle part consists of randomly sampled text tokens, which are sampled from the model's vocabulary and range from 0 to 49405, thus forming a diverse text input. For example: [49406, 3545, 334, …, 2346, 3472, 49407].
[0058] Step S3: Encode the text description obtained in step S1 using the text encoder of the visual language model and extract the corresponding text features.
[0059] Step S3 further includes the following steps:
[0060] Step S3-1, using a text encoder based on a visual language model. The text description generated in step S1 Encode the text to obtain its corresponding text features, and then normalize them to obtain the text features as follows: :
[0061]
[0062] Step S4: Use the text encoder of the visual language model to encode the random text obtained in step S2 and extract the corresponding text features.
[0063] Step S4 further includes the following steps:
[0064] Step S4-1, using a text encoder based on a visual language model. For the random text generated in step S2 Encode the text to obtain its corresponding text features, and then normalize them to obtain random text features. :
[0065]
[0066] Among them, symbols Represents the real number field. This indicates the number of random texts generated. This represents the feature dimension of the text encoder output.
[0067] Step S4-2, process the random text features obtained in step S4-1. Perform QR decomposition to obtain an orthogonal matrix. and upper triangular matrix It satisfies:
[0068]
[0069] Among them, symbols Represents the real number field. This indicates the number of random texts generated. This represents the feature dimension of the text encoder output. (Orthogonal matrix) satisfy , Indicates transpose. Represent the identity matrix. And define the orthogonal matrix. As orthogonalized random text features, to ensure that they are consistent with the original random text features Maintain a consistent dimension.
[0070] Step S5: Encode the input image using the image encoder of the visual language model and extract the corresponding image features.
[0071] Step S5 further includes the following steps:
[0072] Step S5-1, clean the example Image encoder for input visual language model The corresponding features are obtained, and the image features are normalized to obtain the image features. :
[0073]
[0074] Step S6: Based on the text description features obtained in step S3 and the image features obtained in step S5, an adversarial example is generated iteratively using the PGD attack algorithm.
[0075] Step S6 further includes the following steps:
[0076] Step S6-1: Based on the text description features obtained in step S3 and the image features obtained in step S5 Calculate cosine similarity :
[0077]
[0078] For the The image features and the first The similarity between the text description features is defined as follows: Each image-text pair corresponds to a one-hot tag. Among them, positive sample pairs = 1, other cases = 0. Given the input Under the condition, output The probability of occurrence can be expressed as :
[0079]
[0080] Among them, parameters This represents the temperature coefficient, used to adjust the smoothness of the probability distribution. Traversing the image All candidate text descriptions. Based on this, calculate the cross-entropy loss:
[0081]
[0082] in This represents the expectation of all image-text pairs, i.e., the average over all samples.
[0083] Step S6-2: Using the cross-entropy loss obtained in step S6-1, the PGD attack algorithm is used to iteratively generate adversarial examples on the input image.
[0084]
[0085] in, Indicates the first Adversarial examples generated in the next iteration Represents the loss function Example of the previous round of confrontation
[0086] The gradient of the sign function. Extract the sign of the gradient to determine the direction of the perturbation without changing its magnitude. Parameters Used to control the intensity of disturbances. This refers to the projection operator, used to restrict the generated adversarial examples to the original examples. of Within the domain, to maintain its similarity to the original sample.
[0087] Step S7: Based on the text description features obtained in step S3 and the image features extracted in step S5 from the adversarial examples in step S6, calculate the cross-entropy loss and perform adversarial training.
[0088] Step S7 further includes the following steps:
[0089] Step S7-1, the adversarial examples obtained in step S6 Input the image encoder of the currently trained model in step S5. Obtain adversarial example image features
[0090]
[0091] and the text description features obtained in step S3 Calculate cosine similarity :
[0092]
[0093] For the The adversarial example image features and the first The similarity between the text description features is defined as follows: Each image-text pair corresponds to a one-hot tag. Among them, positive sample pairs = 1, other cases =0. Given the input... Under the condition, output The probability of occurrence can be expressed as :
[0094]
[0095] Among them, parameters This represents the temperature coefficient, used to adjust the smoothness of the probability distribution. Traversal of the opponent's example All candidate text descriptions. Based on this, calculate the cross-entropy loss:
[0096]
[0097] in This represents the expectation of all image-text pairs, i.e., the average over all samples.
[0098] Step S8: Based on the random text features obtained in step S4, calculate the cosine similarity with the adversarial example image features and clean example image features extracted in step S5, and reduce the difference between the two by alignment constraints.
[0099] Step S8 further includes the following steps:
[0100] Step S8-1: Calculate the orthogonal matrix of the random text features obtained in step S4. The adversarial example obtained in step S6 Input the image encoder of the currently trained model in step S5. Extracted image features Cosine similarity between :
[0101]
[0102]
[0103] Step S8-2: Calculate the random text features obtained in step S4. and clean examples Input the unadversarially trained base model in step S5. Extracted image features Cosine similarity between :
[0104]
[0105]
[0106] Step S8-3: Apply distance constraints to the two cosine similarities obtained in steps S8-1 and S8-2, using a loss function. Achieve alignment between adversarial example features and clean example features in the semantic space of random text:
[0107]
[0108] Indicates the first An example of confrontation, Indicates the first A clean example This indicates the quantity of the two features.
[0109] Step S9: Based on the random text features obtained in step S4, calculate the cosine similarity between the random text features and the clean examples extracted in step S5 in different models, and apply alignment constraints to the two.
[0110] Step S9 further includes the following steps:
[0111] Step S9-1: Calculate the random text features obtained in step S4. And the clean example extracted in step S5 In the base model without adversarial training Image features Cosine similarity between :
[0112]
[0113]
[0114] Step S9-2: Calculate the random text features obtained in step S4. And the clean example extracted in step S5 In the current training model Image features Cosine similarity between :
[0115]
[0116]
[0117] Step S9-3: Apply distance constraints to the two cosine similarities obtained in steps S9-1 and S9-2, using a loss function. Maintaining feature consistency and generalization ability of the model under zero-shot conditions:
[0118]
[0119] Indicates the first A clean example This indicates the quantity of the two features.
[0120] Finally, the model is updated using the total loss:
[0121]
[0122] in, The loss is calculated in step S7. and These are the losses obtained in steps S8 and S9, respectively. and The balancing parameters are 0.08 and 0.05, respectively.
[0123] The method for zero-shot inference using the above model is as follows:
[0124] First, the parameters of the trained visual language model are loaded. During the zero-shot inference phase, the image to be tested is input into the model's image encoder, and the corresponding text prompts for each category are input into the model's text encoder. Image features and text features are extracted separately. The model calculates the similarity between the image features and the text features of each category, sorts them according to the similarity, and finally uses the category with the highest similarity as the prediction result for the image.
[0125] The detailed descriptions listed above are merely specific descriptions of feasible embodiments of the present invention, and are not intended to limit the scope of protection of the present invention. All equivalent methods or modifications that do not depart from the technology of the present invention should be included within the scope of protection of the present invention.
Claims
1. A zero-shot adversarial robust model based on text information augmentation, characterized in that, The model is obtained through the following steps: S1 uses prompts combined with a large model to generate corresponding appearance descriptions for each category, and merges them with category prompts to form a more comprehensive text description; S2, Random text containing vocabulary information is randomly generated based on the model's vocabulary; S3, use the text encoder of the visual language model to encode the text description obtained in step S1 and extract the corresponding text description features. S4. Use the text encoder of the visual language model to encode the random text obtained in step S2 and extract the corresponding random text features. S5 uses an image encoder based on a visual language model to encode the input image and extract corresponding image features; S6. Based on the text description features obtained in step S3 and the clean example image features obtained by the method in step S5, adversarial examples are generated iteratively using the PGD attack algorithm. S7. Based on the text description features obtained in step S3 and the adversarial example image features extracted in step S5 from the adversarial example in step S6, calculate the cross-entropy loss and perform adversarial training. S8. Based on the random text features obtained in step S4, calculate the cosine similarity with the adversarial example image features and the clean example image features extracted according to the method in step S5, and reduce the difference between the two by alignment constraints. S9. Based on the random text features obtained in step S4, calculate the cosine similarity with the image features of the clean examples extracted according to the method in step S5 in different models, and maintain the generalization performance of the model under zero-sample conditions by alignment constraints.
2. The zero-shot adversarial robust model based on text information enhancement according to claim 1, characterized in that, The specific implementation of step S1 is as follows: Step S1-1: Use prompts to generate detailed appearance descriptions of the corresponding categories in the large model, and filter the length of the generated text to ensure that it meets the maximum text input length requirement of the model. Step S1-2: Combine and merge the detailed appearance description of the category generated in step S1-1 with the original category hints to obtain the final text description.
3. The zero-shot adversarial robust model based on text information enhancement according to claim 2, characterized in that, The specific implementation of step S2 is as follows: Based on the token sequence of the model vocabulary, a preset number of random texts are generated for each training batch to construct text inputs with diverse semantic distributions; Step S3 further includes: normalizing the extracted text description features.
4. The zero-shot adversarial robust model based on text information enhancement according to claim 3, characterized in that, The specific implementation of step S4 is as follows: Step S4-1: Encode the text encoder of the random text input visual language model generated in step S2 to obtain its corresponding random text features, and normalize the random text features. Step S4-2: Apply QR decomposition to the random text features obtained in step S4-1 to achieve feature orthogonalization and ensure that the resulting orthogonal matrix and the random text features maintain the same dimension.
5. The zero-shot adversarial robust model based on text information enhancement according to claim 4, characterized in that, The specific implementation of step S6 is as follows: Step S6-1: Calculate the cosine similarity and cross-entropy loss based on the text description features obtained in step S3 and the clean example image features obtained according to the method in step S5. Step S6-2: Based on the cross-entropy loss obtained in step S6-1, the PGD attack algorithm is used to iteratively generate adversarial examples from the input image.
6. The zero-shot adversarial robust model based on text information enhancement according to claim 5, characterized in that, The specific implementation of step S7 is as follows: The adversarial example obtained in step S6 is used to extract adversarial example image features through the image encoder described in step S5, and cosine similarity is calculated with the text description features obtained in step S3, and then cross-entropy loss is calculated to train the model.
7. The zero-shot adversarial robust model based on text information enhancement according to claim 6, characterized in that, The specific implementation of step S8 is as follows: Step S8-1: Calculate the cosine similarity between the random text features obtained in step S4 and the adversarial example image features obtained in step S6 according to the method in step S5. Step S8-2: Calculate the cosine similarity between the random text features obtained in step S4 and the clean example image features obtained by the clean example in step S5. Step S8-3: Apply distance constraints to the two cosine similarities obtained in steps S8-1 and S8-2 to achieve alignment between adversarial example features and clean example features in the random text semantic space.
8. The zero-shot adversarial robust model based on text information enhancement according to claim 7, characterized in that, The specific implementation of step S9 is as follows: Step S9-1: Calculate the cosine similarity between the random text features obtained in step S4 and the clean examples extracted in step S5 in the image features of the base model that has not been trained adversarially. Step S9-2: Calculate the cosine similarity between the random text features obtained in step S4 and the clean examples extracted in step S5 in the image features of the current training model. Step S9-3 applies distance constraints to the two cosine similarities obtained in steps S9-1 and S9-2 to maintain the model's feature consistency and generalization ability under zero-sample conditions.
9. The inference method for a zero-shot adversarial robust model based on text information enhancement according to any one of claims 1-8, characterized in that, In the zero-shot inference stage, the image to be tested is input into the model's image encoder, and the corresponding category of text prompts is input into the model's text encoder to extract image features and text features respectively. The model calculates the similarity between image features and text features of each category, sorts them according to the similarity, and finally uses the category with the highest similarity as the prediction result for the image.
10. A computer device, characterized in that, The computer device loads the trained visual language model parameters of claim 8 and performs prediction according to the inference method of claim 9.