Multi-modal large model confrontation safety detection system and method

By designing an adversarial sample generation and detection system in a multimodal large model, the problem of adversarial safety detection in non-white box scenarios is solved, the generation and evaluation of multimodal coordinated adversarial samples is realized, and the multimodal large model is improved. Confrontational safety detection capability.

CN118916833AActive Publication Date: 2024-11-08ZHEJIANG UNIV

Patent Information

Application Number
CN202410952564.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-08
Estimated Expiration
2044-07-16

AI Technical Summary

Technical Problem

The prior art is difficult to effectively detect the adversarial security of multimodal large models in non-white box scenarios, and traditional methods ignore the synergistic effects between multimodal data and cannot fully detect the adversarial security performance of multimodal large models.

Method used

A multimodal large-modal model adversarial security detection system was designed, and through parameter settings modules, image adversarial sample generation modules and text adversarial sample generation modules are generated to generate adversarial samples of image and text modality, and combined with multimodal task testing modules. Conduct adversarial training and evaluation, fully consider the interactions between various modes, and construct multimodal coordinated adversarial samples.

Benefits of technology

It realizes multimodal adversarial security detection without the white box model, can perform security assessment in non-white box scenarios, supports the generation of adversarial samples of a variety of multimodal data, and improves the accuracy and scalability of adversarial security detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916833A_ABST
    Figure CN118916833A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-modal large model confrontation safety detection system and method, and aims to evaluate the robustness and confrontation safety of a multi-modal large model. According to the method, the input data is processed through the single-mode encoder, and the adversarial disturbance is applied to the obtained embedded representation to weaken the collaborative consistency among the multi-mode data, so that the multi-mode adversarial sample is constructed and is used for detecting the accuracy of the multi-mode large model when the multi-mode large model executes a downstream task. The method is suitable for a non-white-box test scene, the difficulty of model security assessment is effectively reduced, and the universality of assessment is improved; the proposed multi-modal confrontation sample collaborative generation scheme can effectively utilize the potential relation among the modal data to realize more comprehensive confrontation security detection; in addition, the method has good expandability, and supports generation of adversarial samples of various multi-modal data including texts and images, texts and audios, texts and silent videos and the like.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence security, and in particular to a multimodal large model adversarial security detection system and method. Background Art

[0002] With the rapid development of deep learning technology in recent years, large language models (hereafter referred to as large models) have demonstrated strong application potential in many fields. From initial text generation to the current fusion of multimodal information such as speech and audio, large models have become a research hotspot in the field of artificial intelligence.

[0003] Initial large-scale models focused primarily on text interactions, aiming to address natural language processing tasks such as machine translation, text summarization, and question-answering systems. As technology advances, researchers are exploring ways to incorporate multimodal information into large-scale models to enhance their generalization capabilities in real-world scenarios. Multimodal large-scale models combine language processing capabilities with the ability to analyze and generate diverse information modalities. They can process not only text but also multimodal data such as audio and video, enabling richer and more natural interactions.

[0004] However, while large multimodal models are rapidly developing, their security concerns cannot be ignored. Adversarial security has become a major focus of research. Adversarial attacks involve adding small malicious perturbations (called adversarial examples) to input data, causing the model to produce incorrect responses, thereby impacting the accuracy of downstream tasks performed by the large model. When models make errors in critical downstream tasks, especially in safety-critical applications, this can pose potential security risks. Furthermore, frequent model errors can undermine public trust in the model. For example, in visual question-answering tasks, where users use intelligent assistants to solve problems, if the model provides irrelevant answers or incorrect information, it can mislead users into making incorrect decisions. In visual entailment tasks, where models must reason based on visual information, such as diagnosing medical conditions using medical images, inaccurate predictions can have serious consequences. Compared to traditional large text generation models, large multimodal models offer many advantages in terms of adversarial security. On the one hand, the presence of multimodal information allows attackers to exploit a wider variety of data, increasing the complexity and effectiveness of attacks. On the other hand, during the training and inference of large multimodal models, data from different modalities may influence each other, making the model more vulnerable to attacks. Therefore, studying the security issues of large multimodal models has important theoretical and practical value.

[0005] Currently, adversarial security research on large multimodal models is mostly conducted on white-box models. However, some large models are still in the stage where the code is not publicly available. Therefore, conducting security assessments in non-white-box scenarios has become a research focus. The core idea of ​​large multimodal models is to combine the large model with encoders of one or more input types (such as images, audio, silent videos, etc.). The encoders output embedded representations of the data in each modality and provide them to the back-end large model core. Since the structural information of various encoders is generally public, a natural solution is to construct adversarial samples for model security testing by perturbing the encoder output embedded representations based on only the information of the front-end multimodal encoder.

[0006] Secondly, generating effective adversarial test samples is also a challenge. The simplest approach to multimodal adversarial testing is to directly combine adversarial text and adversarial images to construct a combination of adversarial samples. Due to the discrete nature of text, which is difficult to align with data from modalities like images and audio, most studies have adopted a "text-first" attack strategy, identifying the text attack first and then training adversarial samples from other modalities. This approach ignores the synergy between modalities and cannot fully test the adversarial security performance of large multimodal models. Therefore, there is an urgent need to explore adversarial security detection strategies that can fully leverage the bidirectional relationships between multimodal models.

[0007] In summary, there is an urgent need for a multimodal collaborative adversarial security detection solution that does not require a white-box model to ensure the security of large multimodal models in practical applications. Summary of the Invention

[0008] The purpose of the present invention is to provide a multimodal large model adversarial security detection system and method to address the deficiencies of the existing technology.

[0009] The object of the present invention is achieved through the following technical solutions:

[0010] A multimodal large-model adversarial security detection system, comprising:

[0011] The parameter setting module is used to set the parameters needed in the detection process, including the multimodal large model to be detected, the type of encoder for each modality used by the model, the number of image adversarial sample training cycles, the number of text adversarial sample candidates, and the downstream tasks of the multimodal large model;

[0012] Image adversarial sample generation module, used for adversarial training to generate adversarial samples of image modality;

[0013] Text adversarial sample generation module, used to generate adversarial samples in text mode based on BERT-Attack;

[0014] The multimodal task testing module is used to test the selected multimodal downstream task on the selected multimodal large model, and finally obtain the accuracy of the model on the task.

[0015] Furthermore, the modal encoder types include image encoders and text encoders; and the multimodal data include original image samples and original text samples.

[0016] Furthermore, the parameter setting module provides parameters to the image adversarial sample generation module, the text adversarial sample generation module and the multimodal task testing module; at the same time, the system automatically loads the multimodal data and downstream tasks required for detection for subsequent adversarial sample generation and task testing.

[0017] Furthermore, the image adversarial sample generation module accepts multimodal data, each modality encoder and the number m of image adversarial sample training cycles from the parameter setting module; and obtains the image adversarial sample through adversarial training.

[0018] Furthermore, the text adversarial sample generation module accepts multimodal data, each modality encoder and the number of text adversarial sample candidates n from the parameter setting module; obtains a text adversarial sample library of size n based on BERT-Attack, and obtains text adversarial samples according to the scoring criteria.

[0019] Furthermore, the multimodal task testing module accepts the image adversarial samples obtained by the image adversarial sample generation module and the text adversarial samples obtained by the text adversarial sample generation module, and inputs them into a large multimodal model to be detected for a downstream task to obtain the detection accuracy.

[0020] A multimodal large model adversarial security detection method is applied to the above-mentioned multimodal large model adversarial security detection system, and the method comprises the following steps:

[0021] (1) Select the multimodal large model to be tested, confirm the type of multimodal encoder used, set the number of image adversarial sample training cycles m, the number of text adversarial sample candidates n, and select the large model task for security testing; click the Start Test button, and the system automatically loads the multimodal data and downstream tasks required for detection;

[0022] (2) Inputting the original image sample, the original text sample, the image encoder, the text encoder, and the number of image adversarial sample training cycles m into the image adversarial sample generation module;

[0023] First, the original image sample is input into the image encoder to obtain the original image embedding representation, and the original text sample is input into the text encoder to obtain the original text embedding representation. Gaussian noise is added to the original image embedding representation, and adversarial training is performed within the number of cycles m to obtain the image adversarial sample.

[0024] (3) inputting the image adversarial sample, the original text sample, the image encoder, the text encoder, and the number of text adversarial sample candidates n into a text adversarial sample generation module; the text adversarial sample generation module generates n text adversarial samples according to the perturbation degree from small to large; and selecting the text adversarial sample with the highest score;

[0025] (4) Combining the image adversarial sample obtained in step (2) and the text adversarial sample obtained in step (3) into a multimodal adversarial sample, and inputting the sample into a multimodal task testing module; outputting the detection process information on the task and the final accuracy rate in the system.

[0026] Furthermore, in step (2), the distance between the image adversarial embedding representation and the image original embedding representation, as well as the distance between the image adversarial embedding representation and the text original embedding representation are maximized, as shown in formula (1):

[0027] max L(E i (x′ i ),E i (x i ))+α·L(E i (x′ i ),E t (x t )) (1)

[0028] Where L(·) represents the distance function between embedded representations, i.e., KL divergence loss function; E i (·) and E t (·) represent the image encoder and text encoder respectively, x i and x′ i Represent the original image sample and the image adversarial sample, respectively, x t Represents the original text sample, and α is a fixed scale coefficient.

[0029] Furthermore, in step (3), the text adversarial sample generation module sets the perturbation degree to 1; based on the original text sample and the text encoder, a sequence of key words in the original text sample that will affect the decision-making of the large model is obtained, and substitute words are generated for these words respectively; then, the substitute words are used to replace the keywords in sequence to generate text adversarial samples until the number of words that are different between the text adversarial sample and the original text sample reaches a preset perturbation degree; the obtained text adversarial samples are counted in the text adversarial sample library, the perturbation degree is increased by 1, and the above process is continued until the library contains n text adversarial samples; each text adversarial sample in the text adversarial sample library is scored, and the scoring formula is shown in formula (2):

[0030] score=β·L(E t(x′ t ),E i (x′ i ))+(β-1)·L(E t (x′ t ),E t (x t )) (2)

[0031] Where L(·) represents the distance function between embedded representations, i.e., KL divergence loss function; E i (·) and E t (·) represent the image encoder and text encoder respectively, x t and x′ t Represent the original text sample and the text adversarial sample, x′ i represents the image adversarial sample, β is a fixed scaling coefficient, and the text adversarial sample with the highest score is selected.

[0032] The beneficial effects of the present invention are:

[0033] 1. Non-white-box evaluation scheme: This paper proposes a multimodal large-model adversarial security detection method that does not require a white-box scenario. This method only requires understanding the structural information of the front-end multimodal encoder and perturbing the embedded representation generated by the encoder to generate adversarial samples for security assessment.

[0034] 2. Multimodal adversarial sample collaborative generation scheme: This invention fully considers the interaction relationship between various modalities and designs an adversarial sample generation scheme that combines multimodal data. It can construct multimodal collaborative adversarial samples, thereby carrying out more comprehensive multimodal large-scale model adversarial security evaluation.

[0035] 3. Scalability: This invention has good scalability and supports the generation of adversarial samples for various multimodal data types, including text and images, text and audio, and text and silent video. Technical implementation requires only replacing the corresponding multimodal encoder. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is an architectural diagram of the present invention;

[0037] Figure 2 It is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0038] The present invention will be described in detail below with reference to the accompanying drawings. Unless there is any conflict, the features of the following embodiments and implementations may be combined with each other.

[0039] The architecture diagram of the present invention is as follows Figure 1 As shown, the method flow is as follows Figure 2As shown in the figure, the user selects the multimodal large model to be tested, confirms the type of multimodal encoder used, sets relevant key parameters, and selects the large model task for which security testing is required. Clicking the Start Test button displays the system's detection process information for the selected multimodal large model for the selected task, along with the final accuracy, reflecting the model's adversarial security performance.

[0040] Deployed in non-white-box scenarios, this method does not require access to the internal structure or algorithmic details of large models. It only requires the structural information of the front-end multimodal encoder. It generates multimodal adversarial examples by perturbing the embedded representation of the input data after encoder processing. Furthermore, this method considers the interactions between the various modalities and designs a multimodal adversarial example construction strategy to enable adversarial security assessment of large multimodal models.

[0041] See also Figure 1 The present invention provides a multimodal large model adversarial security detection system, comprising:

[0042] The parameter setting module is used to set the parameters required for the detection process, including the multimodal large model to be tested, the encoder types used by each modality, the number of image adversarial sample training cycles (m), the number of text adversarial sample candidates (n), and the downstream tasks for the multimodal large model. This module provides these parameters to the subsequent image adversarial sample generation module, text adversarial sample generation module, and multimodal task testing module. The system also automatically loads the multimodal data and downstream tasks required for detection, which are then used for adversarial sample generation and task testing.

[0043] The image adversarial sample generation module is used to perform adversarial training to generate adversarial samples in the image modality. This module accepts multimodal data (original image samples, original text samples), encoders for each modality (image encoder, text encoder), and the number of image adversarial sample training cycles m; it generates image adversarial samples through adversarial training.

[0044] The Text Adversarial Example Generation module is used to generate adversarial examples in the text modality based on BERT-Attack. This module accepts multimodal data (image adversarial examples, original text examples), encoders for each modality (image encoder, text encoder), and the number of text adversarial example candidates n. It then generates a text adversarial example library of size n based on BERT-Attack and generates text adversarial examples based on a scoring criterion.

[0045] The multimodal task testing module is used to test the selected multimodal large model on a selected multimodal downstream task, ultimately determining the model's accuracy on that task. This module accepts image adversarial samples generated by the image adversarial sample generation module and text adversarial samples generated by the text adversarial sample generation module, and inputs them into the multimodal large model to be tested for a specific downstream task to determine the detection accuracy.

[0046] The present invention also provides a multi-modal large model adversarial security detection method, which is applied to the above-mentioned multi-modal large model adversarial security detection system. The method includes the following steps:

[0047] Step 1: Select the multimodal large model to be tested, confirm the type of multimodal encoder used, set the number of image adversarial sample training cycles (m) and the number of text adversarial sample candidates (n), and select the large model task for security testing. Click the Start Test button. The system automatically loads the multimodal data and downstream tasks required for testing.

[0048] Assume that the ALBEF multimodal model is selected, using the ViT-B / 16 image encoder and the BERT text encoder, setting the number of adversarial example training cycles (m) to 10 and the number of text adversarial example candidates (n) to 10, for the visual entailment (VE) task. Click the Start button. The multimodal data required for the test task will automatically load in the backend.

[0049] Step 2: The original image sample, original text sample, image encoder, text encoder, and image adversarial sample training cycle number m are input into the image adversarial sample generation module. First, the original image sample is input into the image encoder to obtain the original image embedding representation, and the original text sample is input into the text encoder to obtain the original text embedding representation. Gaussian noise is added to the original image embedding representation, and adversarial training is performed within the number of cycles m. The goal of adversarial training is to maximize the distance between the image adversarial embedding representation and the original image embedding representation, as well as the distance between the image adversarial embedding representation and the original text embedding representation, as shown in Equation 1:

[0050] max L(E i (x′ i ),E i (x i ))+α·L(E i (x′ i ),E t (x t )) (1)

[0051] Where L(·) represents the distance function between embedded representations, which is the KL divergence loss function here; E i (·) and E t(·) represent the image encoder and text encoder respectively, x i and x′ i Represent the original image sample and the image adversarial sample, respectively, x t Represents the original text sample, α is a fixed scale factor, which is 3 here.

[0052] Step 3: The image adversarial sample, original text sample, image encoder, text encoder, and the number of text adversarial sample candidates n are input into the text adversarial sample generation module. This module generates n text adversarial samples from small to large according to the perturbation degree (i.e., the number of words that differ between the text adversarial sample and the original text sample). First, the text adversarial sample generation module sets the perturbation degree to 1. Then, based on the original text sample and the text encoder, it obtains the key word sequence in the original text sample that affects the decision-making of the large model and generates replacement words for these words. Then, it sequentially replaces the key words with the replacement words to generate text adversarial samples until the number of words that differ between the text adversarial sample and the original text sample reaches the pre-set perturbation degree. The resulting text adversarial samples are counted in the text adversarial sample library, and the perturbation degree is increased by 1. The above process continues until the library contains n text adversarial samples. The text adversarial sample generation module can be implemented using BERT-Attack technology (Li L, Ma R, Guo Q, et al. Bert-attack: Adversarial attack against bert using bert[J]. arXiv preprint arXiv:2004.09984, 2020.). Afterwards, each text adversarial sample in the text adversarial sample library is scored, and the scoring formula is shown in Equation 2:

[0053] score=β·L(E t (x′ t ),E i (x′ i ))+(β-1)·L(E t (x′ t ),E t (x t )) (2)

[0054] Where L(·) represents the distance function between embedded representations, which is the KL divergence loss function here; E i (·) and E t (·) represent the image encoder and text encoder respectively, x t and x′ t Represent the original text sample and the text adversarial sample, x′ i Denotes an image adversarial sample, β is a fixed scaling factor, here 0.7. Select the text adversarial sample with the highest score.

[0055] Step 4: Combine the image adversarial sample obtained in step 2 and the text adversarial sample obtained in step 3 into a multimodal adversarial sample and input it into the multimodal task testing module. The system outputs the detection process information on this task and the final accuracy.

[0056] The above embodiments are intended only to illustrate the design concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. The scope of protection of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design concepts disclosed in the present invention are within the scope of protection of the present invention.

Claims

1. A multi-modal large model adversarial security detection system, characterized in that: include: The parameter setting module is used to set the parameters needed in the detection process, including the multimodal large model to be detected, the type of encoders for each modality used by the model, the number of image adversarial sample training cycles, the number of text adversarial sample candidates, and the downstream tasks of the multimodal large model; Image adversarial sample generation module, used to perform adversarial training to generate adversarial samples of image modality; Text adversarial sample generation module, used to generate adversarial samples in text mode based on BERT-Attack; The multimodal task testing module is used to test the selected multimodal downstream task on the selected multimodal large model, and finally obtain the accuracy of the model on the task.

2. The multimodal large model adversarial security detection system according to claim 1, characterized in that: The modal encoder types include image encoders and text encoders; the multimodal data include original image samples and original text samples.

3. The multimodal large model adversarial security detection system according to claim 1, characterized in that: The parameter setting module provides parameters to the image adversarial sample generation module, the text adversarial sample generation module and the multimodal task testing module; at the same time, the system automatically loads the multimodal data and downstream tasks required for detection for subsequent adversarial sample generation and task testing.

4. The multimodal large model adversarial security detection system according to claim 3, characterized in that: The image adversarial sample generation module receives multimodal data, encoders of each modality and the number of image adversarial sample training cycles m from the parameter setting module; and obtains image adversarial samples through adversarial training.

5. The multimodal large model adversarial security detection system according to claim 4, characterized in that: The text adversarial sample generation module accepts multimodal data, encoders of each modality and the number n of text adversarial sample candidates from the parameter setting module; obtains a text adversarial sample library of size n based on BERT-Attack, and obtains text adversarial samples according to scoring criteria.

6. The multimodal large model adversarial security detection system according to claim 5, characterized in that: The multimodal task testing module accepts the image adversarial samples obtained by the image adversarial sample generation module and the text adversarial samples obtained by the text adversarial sample generation module, and inputs them into a multimodal large model to be detected for a downstream task to obtain the detection accuracy.

7. A multimodal large model adversarial security detection method, applied to the multimodal large model adversarial security detection system according to any one of claims 1 to 6, characterized in that: The method comprises the following steps: (1) Select the multimodal large model to be tested, confirm the type of multimodal encoder used, set the number of image adversarial sample training cycles m, the number of text adversarial sample candidates n, and select the large model task for security testing; click the Start Test button, and the system automatically loads the multimodal data and downstream tasks required for detection; (2) inputting the original image sample, the original text sample, the image encoder, the text encoder, and the number of image adversarial sample training cycles m into the image adversarial sample generation module; First, the original image sample is input into the image encoder to obtain the original image embedding representation, and the original text sample is input into the text encoder to obtain the original text embedding representation; Gaussian noise is added to the original image embedding representation, and adversarial training is performed within the number of cycles m to obtain image adversarial samples; (3) inputting the image adversarial sample, the original text sample, the image encoder, the text encoder, and the number of candidate text adversarial samples n into a text adversarial sample generation module; the text adversarial sample generation module generates n text adversarial samples according to the perturbation degree from small to large; and selecting the text adversarial sample with the highest score; (4) Combining the image adversarial sample obtained in step (2) and the text adversarial sample obtained in step (3) into a multimodal adversarial sample, and inputting the sample into a multimodal task testing module; outputting the detection process information on the task and the final accuracy rate in the system.

8. The multimodal large model adversarial security detection method according to claim 7 is characterized in that: In step (2), the distance between the image adversarial embedding representation and the image original embedding representation, as well as the distance between the image adversarial embedding representation and the text original embedding representation are maximized, as shown in formula (1): max L(E i (x′ i ),It i (x i ))+α·L(E i (x′ i ),It t (x t )) (1) Where L(·) represents the distance function between embedded representations, i.e., KL divergence loss function; E i (·) and E t (·) represent the image encoder and text encoder respectively, x i and x′ i Represent the original image sample and the image adversarial sample, respectively, x t Represents the original text sample, and α is a fixed scale factor.

9. The multimodal large model adversarial security detection method according to claim 7, characterized in that: In the step (3), the text adversarial sample generation module sets the perturbation degree to 1; based on the original text sample and the text encoder, obtain the key word sequence in the original text sample that will affect the decision-making of the large model, and generate replacement words for these words respectively; then use the replacement words to replace the keywords in turn to generate text adversarial samples until the number of words different between the text adversarial sample and the original text sample reaches a preset perturbation degree; count the obtained text adversarial samples into the text adversarial sample library, increase the perturbation degree by 1, and continue the above process until the library contains n text adversarial samples; score each text adversarial sample in the text adversarial sample library, and the scoring formula is shown in formula (2): score=β·L(E t (x′ t ),AND i (x′ i ))+(β-1)·L(E t (x′ t ),AND t (x t )) (2) Where L(·) represents the distance function between embedded representations, i.e., KL divergence loss function; E i (·) and E t (·) represent the image encoder and text encoder respectively, x t and x′ t Represent the original text sample and the text adversarial sample, respectively, x′ i represents the image adversarial sample, β is a fixed scaling coefficient, and the text adversarial sample with the highest score is selected.

Citation Information

Patent Citations

  • Vision-language pre-training model-oriented multi-modal migration countermeasure attack method

    CN117094000A

  • Post-fusion camouflage method and device based on cross contrast learning

    CN118247598A

  • Systems and methods for vision-and-language representation learning

    US20220391755A1

  • Detecting adversarial examples using latent neighborhood graphs

    US20230334332A1

Cited By

  • Safety compliance evaluation system and method based on multi-modal large model

    CN120930150A

  • An end-side large model security evaluation method based on physical signal disturbance

    CN122433091B