Multi-modal large model adversarial sample detection method based on attention migration

Abstract

The invention discloses a multi-modal large model confrontation sample detection method based on attention migration, and aims to solve the problems of high confrontation training defense cost and poor confrontation detection performance in the prior art by deeply analyzing the internal attention mechanism of a multi-modal large model. The attention distribution difference of the adversarial sample and the benign sample between the user instruction and the multi-modal data is stably described, so that reliable distinguishing of the adversarial sample and the benign sample is realized; the method only depends on the existing attention weight information in the model reasoning process, the lightweight feature dimension reduction and the simple binary classifier are combined for detection, an extra large-scale model does not need to be introduced or adversarial training and parameter fine adjustment do not need to be carried out, and the method has the advantages of being low in calculation overhead and deployment cost; the invention provides a non-intrusive and extensible large model reasoning stage confrontation security defense scheme, and the reasoning security and robustness of the multi-mode large model in a confrontation environment can be effectively improved on the premise of not influencing the original functions of the multi-mode large model.

Description

Technical Field

[0001] This invention pertains to adversarial security defense in the model inference stage of computer artificial intelligence security, specifically involving a multimodal large model adversarial sample detection method based on attention shift. Background Technology

[0002] In recent years, with the rapid development of multimodal understanding and generation technologies, large multimodal models have made breakthroughs in speech understanding, image understanding, cross-modal reasoning, and complex task execution, and have been widely applied in scenarios such as intelligent assistants, human-computer interaction, and automatic content generation. Large multimodal models typically integrate multiple input interfaces such as text, speech, and images. While their functionality has been significantly enhanced, this has also introduced more complex and covert adversarial security issues. Existing research shows that adversarial examples targeting large multimodal models have become a key technological foundation for various security threats such as hint injection attacks, model jailbreaking attacks, and model theft attacks. Attackers can induce unexpected behavior in models without altering their structure and parameters through carefully crafted speech, image, or cross-modal inputs, seriously threatening the security and reliability of the model's inference phase.

[0003] To ensure the inference security of large multimodal models in adversarial environments, adversarial defense and detection technologies are of significant practical importance. However, existing methods still have obvious shortcomings: on the one hand, large multimodal models have a large number of parameters, and using adversarial training or fine-tuning requires high computational and storage costs, making it difficult to promote in practical deployments; on the other hand, large multimodal models usually follow an autoregressive text generation paradigm and employ decoding strategies such as random sampling during the inference stage, resulting in strong uncertainty in their output text, making it difficult to extract stable and repeatable features from the generation results for adversarial sample detection.

[0004] Therefore, how to discover highly distinguishable and stable adversarial example detection factors without modifying model parameters, and design lightweight and efficient detection methods to provide reliable inference stage security for multimodal large models, has become an urgent technical problem to be solved. Summary of the Invention

[0005] This invention addresses the adversarial security vulnerabilities of existing multimodal large models by proposing an attention-shift-based method for detecting adversarial examples in multimodal large models. This method leverages the interpretability of the attention mechanism within the multimodal large model, analyzes the differences in attention distribution between the model and user commands and multimodal data during inference, constructs highly discriminative adversarial example detection factors, and combines feature dimensionality reduction and a support vector machine classifier to achieve stable and efficient adversarial example detection in multimodal large models.

[0006] The technical solution of the present invention is as follows:

[0007] This invention discloses a multimodal large model adversarial example detection method based on attention shift, comprising:

[0008] Multimodal data and user commands are input into a large multimodal model for inference. After autoregressive generation and random sampling decoding, the response text sequence is obtained.

[0009] During the multimodal large model inference process, attention weights are calculated layer by layer and head by head for generated lexical units to multimodal data lexical units and user instruction lexical units in the response text sequence;

[0010] Based on attention weights, the layer-head attention distribution of generated lexical units on multimodal data and user commands is calculated respectively, and attention shift features are constructed.

[0011] The attention-shifted features are subjected to dimensionality reduction processing to obtain low-dimensional feature vectors;

[0012] Binary classification prediction is performed based on low-dimensional feature vectors, and the output multimodal data consists of detection results for adversarial or benign samples.

[0013] As a further improvement, the multimodal data described in this invention includes voice data or image data, and the multimodal data is either an unattacked benign sample or an adversarial sample constructed by an attacker.

[0014] As a further improvement, the attention weights for calculating the generated lexical units to multimodal data lexical units and user instruction lexical units in the response text sequence layer by layer and head by head, as described in this invention, include:

[0015] In the process of autoregressive generation of multimodal large models, the attention weight matrix corresponding to each layer and each attention head in the backbone network of the model is calculated.

[0016] The generated lexical units are used as query lexical units, and the multimodal data lexical units and user instruction lexical units are used as key lexical units. The corresponding attention weight values ​​are extracted from the attention weight matrix.

[0017] As a further improvement, the attention weights described in this invention are attention distribution weights normalized by Softmax.

[0018] As a further improvement, the constructed attention shift feature described in this invention includes:

[0019] For each layer of the model And every attention head Calculate the average attention weights of the generated lexical units across all multimodal data lexical units to obtain the layer-head attention matrix of the multimodal data. ;

[0020] For each layer of the model And every attention head Calculate the average attention weights of the generated lexical units across all user command lexical units to obtain the layer-head attention matrix of the user command. ;

[0021] Attention matrix of layer heads and Perform a difference operation and flatten it to obtain the attention shift feature vector.

[0022] As a further improvement, the feature dimensionality reduction processing described in this invention employs principal component analysis, including:

[0023] The attention-biased feature vector is centered.

[0024] Calculate the covariance matrix of the centered eigenvectors and perform eigenvalue decomposition;

[0025] Set the number of principal components, select the directions of that number of principal components as the projection basis, and project the attention-shifted feature vectors into the low-dimensional feature space.

[0026] As a further improvement, the number of principal components in this invention is preferably 2, so as to obtain a two-dimensional feature vector for adversarial sample detection.

[0027] As a further improvement, the binary classification prediction described in this invention uses a support vector machine classifier. The support vector machine is trained based on the features of labeled benign and adversarial samples, and outputs the corresponding category label during the attack detection stage.

[0028] The beneficial effects of this invention are as follows:

[0029] This invention discloses a method for detecting adversarial examples in multimodal large models based on attention shift. This method detects attacks by deeply analyzing and examining the attention distribution within the multimodal large model. During the inference stage of the multimodal large model, the attention weight distribution of generated lexical pairs on multimodal data and user commands is extracted. Multi-layer, multi-head attention weights are aggregated to construct stable attention shift features. Then, distinguishable principal component elements are mined from the high-dimensional attention shift features through feature dimensionality reduction. Finally, adversarial example detection is achieved based on a linear binary classifier. This method does not require modification of the structure or parameters of the multimodal large model; it relies solely on existing attention information from the inference process to complete the detection, effectively avoiding the high costs associated with adversarial training and model fine-tuning. By analyzing the stable differences in the attention distribution within the model, this invention can reliably distinguish between adversarial and benign examples, achieving lightweight and efficient adversarial security protection during the inference stage of multimodal large models.

[0030] This invention addresses the problems of high cost and poor performance in adversarial training defenses in existing systems. By deeply analyzing the internal attention mechanism of multimodal large models, it stably characterizes the difference in attention distribution between adversarial and benign samples in user commands and multimodal data, thereby achieving reliable differentiation between adversarial and benign samples. This invention relies only on the attention weight information already present in the model inference process, combined with lightweight feature dimensionality reduction and a simple binary classifier for detection. It does not require the introduction of additional large-scale models or adversarial training and parameter fine-tuning, and has the advantages of low computational overhead and low deployment cost. This invention provides a non-intrusive and scalable adversarial security defense scheme for the inference stage of large models, which can effectively improve the inference security and robustness of multimodal large models in adversarial environments without affecting their original functions. Attached Figure Description

[0031] Figure 1 This is a flowchart of the process steps of the present invention;

[0032] Figure 2 This is a system framework diagram of the present invention. Detailed Implementation

[0033] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are merely intended to assist those skilled in the art in understanding the principles and knowledge of the present invention, and are not intended to limit the scope of the invention, nor should they be considered as limiting the application scenarios of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various modifications or alterations to the invention, but any variations, changes, and transformations made to the embodiments based on the principles and spirit of the invention also fall within the scope defined by the appended claims. And it is obvious that this specification only uses preferred embodiments as examples, and does not need to exhaustively describe all embodiments.

[0034] A method for detecting adversarial examples in multimodal large models based on attention shift includes the following steps:

[0035] Step S100: Input multimodal data and user commands into the multimodal large model for inference, and obtain the text response after sampling and decoding;

[0036] Step S200: Using the text response as a query term, calculate the attention weights of the term to the multimodal data term and the user instruction term layer by layer and head by head.

[0037] Step S300: Summate the attention corresponding to the multimodal data lexical and the user instruction lexical respectively, and then calculate the average value according to the target lexical to obtain the layer head attention. Subtract the values ​​and flatten them to obtain the attention shift feature.

[0038] Step S400: Perform principal component analysis to reduce the dimensionality of the attention shift features, and obtain a low-dimensional, distinguishable feature vector.

[0039] Step S500: Input the feature vector into the support vector machine for classification and output the adversarial attack detection result.

[0040] In the above technical solution, in step S100, the multimodal data includes the speech data or image data to be analyzed. This data can be either benign samples from normal input or adversarial samples constructed by attackers. The user instruction is the text prompt word corresponding to the task. The multimodal data and the user instruction are jointly input into the multimodal large model, and the model generates a text response through an autoregressive generation mechanism combined with a random sampling decoding strategy.

[0041] In the above technical solution, step S200 includes:

[0042] Step S210: During the autoregressive generation process, calculate the attention weight matrix corresponding to each layer and each attention head of the large model backbone network.

[0043] Step S220: Use the tokens generated in each step as query tokens, and the multimodal data tokens and user command tokens as key tokens, to extract the weights of the corresponding rows and columns in the attention matrix. The attention weights are Softmax normalized weights.

[0044] In the above technical solution, step S300 includes:

[0045] Step S310, for each layer Each attention head Calculate the sum of attention weights for query terms across all multimodal data terms, and calculate the mean for all query terms. The layer-head attention matrix of the multimodal data is obtained. , of which Line number Listed as ;

[0046] Step S320, for each layer Every attention head Calculate the sum of attention weights for query terms across all user command terms, and then calculate the mean for each query term. Obtain the layer head attention matrix of the user command. , of which Line number Listed as ;

[0047] Step S330: Subtract the multimodal data from the layer head attention matrix of the user instruction element by element. The resulting matrix is ​​flattened into a one-dimensional vector and used as an attention shift feature.

[0048] In the above technical solution, step S400 includes:

[0049] Step S410: After centering the attention shift feature vector, calculate its covariance matrix and perform eigenvalue decomposition on the covariance matrix.

[0050] Step S420: Based on the set number of principal components k, select the first k principal component directions as projection bases according to the eigenvalue magnitude, and project the original high-dimensional attention shift features onto the principal component space to obtain a low-dimensional feature vector, so as to retain the main discrimination information and reduce the computational complexity.

[0051] In the above technical solution, in step S500, the dimensionality-reduced feature vector is input into a support vector machine (SVM) classifier for classification prediction. The SVM learns the optimal classification hyperplane based on labeled benign and adversarial sample features. During the attack detection phase, it outputs the corresponding category label based on the spatial location of the input features. The category label indicates whether the current input sample is an adversarial or benign sample.

[0052] like Figure 1 As shown, the multimodal large model adversarial example detection method based on attention shift proposed in this invention includes:

[0053] Obtain the multimodal large model to be detected and its corresponding inference interface, wherein the multimodal large model supports at least joint input of text and speech or text and image, and adopts an autoregressive text generation method in the inference stage;

[0054] Construct an input set for adversarial example detection, including user commands and multimodal data. The multimodal data includes both normal speech or image samples and adversarial examples constructed by attackers to induce the large multimodal model to produce unexpected outputs.

[0055] The user instructions and multimodal data are input into a large multimodal model for inference. After autoregressive generation and random sampling decoding, the response text sequence generated by the model is obtained.

[0056] During model inference, the attention weight matrix of each attention head in each layer of the multimodal large model backbone network is extracted. When generating the response text sequence, the multimodal large model calculates the causal attention weights between the currently generated word and all historical words, generating the corresponding attention matrix. ,in, Indicates query term Keyword Attention weights;

[0057] Let the set of generated terms in the response text sequence be denoted as The word set corresponding to the multimodal data is denoted as The set of lexical units corresponding to user commands is denoted as ;

[0058] For each layer And every attention head The average attention allocation value for generated lexical units to multimodal data lexical units and user instruction lexical units is calculated separately. Specifically, the attention allocation value for multimodal data is defined as follows:

[0059]

[0060] in, Indicates the first Layer Querying keywords in each attention focus Attention weights to keyword elements;

[0061] Similarly, the attention allocation value for user commands is defined as follows:

[0062]

[0063] Based on the above calculations, the layer head attention matrices for multimodal data are constructed respectively. and the layer attention matrix of user instructions The elements in the l-th row and h-th column of the matrix correspond to... and ;

[0064] Perform a difference operation on the layer head attention matrix to obtain the attention offset matrix, and flatten it into a one-dimensional vector in a predetermined order as the attention offset feature vector. Its definition is as follows:

[0065]

[0066] in, This represents the operation of expanding the matrix in the order of layers and attention heads;

[0067] The obtained attention shift feature vector set is subjected to feature dimensionality reduction. Specifically, principal component analysis is used to reduce the dimensionality of the attention shift features, including: centering the feature vectors, calculating the covariance matrix, and performing eigenvalue decomposition on the covariance matrix; setting the number of principal components k, sorting them according to the eigenvalues ​​from largest to smallest, and selecting the top... The principal component directions are used as projection bases, where Preferably, 2; the original attention shift feature vector is projected onto the principal component space to obtain a two-dimensional low-dimensional feature representation, so as to retain the main discrimination information and reduce the computational complexity;

[0068] Based on the dimensionality-reduced feature vectors, a support vector machine (SVM) classifier is constructed for adversarial example detection. The SVM is a linear SVM, which learns the optimal classification hyperplane during the training phase based on the features of labeled benign and adversarial samples. During the attack detection phase, the low-dimensional features corresponding to the sample to be detected are input into the SVM, and the corresponding class label is output according to its position relative to the classification hyperplane in the feature space.

[0069] The category labels include at least two categories: “benign samples” and “adversarial samples”, which are used to indicate whether the current multimodal input contains adversarial attacks against the multimodal large model, thereby realizing the automatic detection of adversarial samples.

[0070] The present invention also proposes a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor, when executing the computer program, implements the steps of the above-described multimodal large model adversarial example detection method based on attention shift.

[0071] The present invention also proposes a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described attention-shift-based multimodal large model adversarial example detection method.

[0072] The present invention also proposes a computer program product, which includes a computer program or instructions, which, when executed by a processor, implement the steps of the above-described multimodal large model adversarial example detection method based on attention shift.

[0073] The multimodal large model adversarial example detection method proposed in this invention, by mining the difference in attention distribution between generated lexical pairs and user commands during the model inference process, can effectively distinguish adversarial examples from benign examples without modifying model parameters or introducing additional complex models, and achieve lightweight and stable adversarial security protection during the multimodal large model inference stage.

[0074] Finally, it should be noted that the above embodiments and descriptions are only used to illustrate the technical solutions of the present invention and not to limit it. Those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the disclosure of the technical solutions of the present invention, and all such modifications and substitutions should be covered within the protection scope of the claims of the present invention.

Claims

1. A method for detecting adversarial examples in multimodal large models based on attention shift, characterized in that, include: Multimodal data and user commands are input into a large multimodal model for inference. After autoregressive generation and random sampling decoding, the response text sequence is obtained. During the multimodal large model inference process, attention weights are calculated layer by layer and head by head for generated lexical units to multimodal data lexical units and user instruction lexical units in the response text sequence; Based on the attention weights, the layer-head attention distribution of generated lexical units on multimodal data and user commands is calculated respectively, and attention shift features are constructed. The attention shift features are subjected to feature dimensionality reduction processing to obtain low-dimensional feature vectors; Based on the low-dimensional feature vector, binary classification prediction is performed, and the multimodal data is output as the detection result of adversarial samples or benign samples.

2. The multimodal large model adversarial example detection method based on attention shift according to claim 1, characterized in that, The multimodal data includes voice data or image data, and the multimodal data is either an unattacked benign sample or an adversarial sample constructed by an attacker.

3. The multimodal large model adversarial example detection method based on attention shift according to claim 1 or 2, characterized in that, The attention weights for calculating the generated lexical units to multimodal data lexical units and user instruction lexical units in the response text sequence layer by layer and head by head include: In the process of autoregressive generation of multimodal large models, the attention weight matrix corresponding to each layer and each attention head in the backbone network of the model is calculated. The generated lexical units are used as query lexical units, and the multimodal data lexical units and user instruction lexical units are used as key lexical units. The corresponding attention weight values ​​are extracted from the attention weight matrix.

4. The multimodal large model adversarial example detection method based on attention shift according to claim 3, characterized in that, The attention weights are the attention distribution weights after Softmax normalization.

5. The multimodal large model adversarial example detection method based on attention shift according to claim 4, characterized in that, The constructed attention shift features include: For each layer of the model And every attention head Calculate the average attention weights of the generated lexical units across all multimodal data lexical units to obtain the layer-head attention matrix of the multimodal data. ; For each layer of the model And every attention head Calculate the average attention weights of the generated lexical units across all user command lexical units to obtain the layer-head attention matrix of the user command. ; For the layer head attention matrix and Perform a difference operation and flatten it to obtain the attention shift feature vector.

6. The multimodal large model adversarial example detection method based on attention shift according to claim 1, 2, 4, or 5, characterized in that, The feature dimensionality reduction process employs principal component analysis, including: The attention-biased feature vector is centered. Calculate the covariance matrix of the centered eigenvectors and perform eigenvalue decomposition; Set the number of principal components, select the directions of that number of principal components as the projection basis, and project the attention-shifted feature vectors into the low-dimensional feature space.

7. The multimodal large model adversarial example detection method based on attention shift according to claim 6, characterized in that, The number of principal components is preferably 2, so as to obtain a two-dimensional feature vector for adversarial sample detection.

8. The multimodal large model adversarial example detection method based on attention shift according to claim 7, characterized in that, The binary classification prediction uses a support vector machine classifier, which is trained based on the features of labeled benign and adversarial samples, and outputs the corresponding class label during the attack detection stage.

Images