Universal indirect prompt injection method for multi-modal large model

Abstract

The invention discloses a universal indirect prompt injection method oriented to a multi-modal large model, which aims at solving the problem that discrete lexical elements and continuous feature mixed modeling in multi-modal input cannot be differentiated, introduces a sampling-based gradient estimation mechanism, replaces discrete lexical element selection and index embedding operation with microprobabilistic sampling and matrix multiplication, and improves the accuracy of the multi-modal input. According to the method, end-to-end adversarial sample optimization is achieved, multi-context expectation optimization and an attention guidance mechanism are combined, attention distribution of a model on multi-modal data in a reasoning stage is controlled in an implicit and explicit mode, the generalization ability of indirect prompt injection under the conditions of unknown user instructions and complex contexts is remarkably improved, and the method is suitable for popularization and application. The method is simultaneously suitable for a continuous representation model, a discrete representation model and a multi-modal large model of a continuous-discrete hybrid architecture, and can stably induce the models to generate target responses in multi-modal scenes such as voice and images, so that indirect prompt injection safety risks faced by the multi-modal large model in a reasoning stage are systematically revealed.

Description

Technical Field

[0001] This invention pertains to model adversarial learning in the field of computer artificial intelligence security, specifically involving a general indirect hint injection method for multimodal large models. Background Technology

[0002] Multimodal large models are increasingly being integrated into various intelligent agent frameworks, endowed with capabilities such as long-term memory management, tool invocation, external environment interaction, and multi-round autonomous decision-making, significantly enhancing their autonomy and influence in complex task scenarios. Against this backdrop, multimodal input channels have become a crucial entry point for attackers to manipulate model behavior, with indirect prompt injection attacks, known for their enhanced stealth, being particularly prominent. Multimodal indirect prompt injection refers to attackers embedding malicious instructions into the speech, images, or other multimodal content that the model needs to parse, rather than providing them directly as explicit text instructions. This causes the model to unintentionally execute implicit instructions pre-set by the attacker while understanding multimodal information. Such attacks typically do not rely on tampering with explicit user input, possessing strong stealth and deceptiveness. Once successful, they can induce multimodal large models to perform unauthorized operations, leak sensitive information, incorrectly invoke tools, or permanently pollute the agent's memory, thus posing serious security risks to deployed systems.

[0003] Existing research has theoretically and empirically verified the feasibility of indirect cue injection attacks on large multimodal models. In the image modality, related work mainly embeds hidden instructions into image content through carefully designed text typography, visual layout, or adversarial perturbations to induce the model to perform unexpected behaviors. In the speech modality, studies have also shown that implicit instructions can be injected into the model without affecting human perception by constructing adversarial audio samples. However, existing methods for indirect cue injection still have significant limitations: First, most existing methods rely on gradient optimization processes and are only applicable to model structures using continuous multimodal lexical representations. For large multimodal models with discrete lexical encoding, discrete word segmentation operations, or hybrid architectures, it is difficult to directly and effectively attack them. Second, because large multimodal models are highly sensitive to the context of user instructions, existing attacks often rely on specific or known context settings. When faced with unknown or changing user instructions, their attack effectiveness and stability decrease significantly, and their generalization ability is insufficient.

[0004] Therefore, designing a general indirect hint injection method that can be applied to continuous representation models, discrete representation models, and hybrid architecture models under the condition of only accessing multimodal data, and further improving the stability and generalization ability of such methods under unknown user instruction contexts, has become an important issue restricting the effectiveness of indirect hint injection attacks in real-world application scenarios. Solving this problem is of great significance for comprehensively assessing the security risks faced by large multimodal models in real-world adversarial environments. Summary of the Invention

[0005] This invention addresses the shortcomings of existing multimodal indirect suggestion injection research in terms of versatility and generalization ability, proposing a general indirect suggestion injection method for large multimodal models. Based on a sampling gradient estimation mechanism, this method replaces the non-differentiable discrete word mapping and embedding index operations in large multimodal models with differentiable probability sampling and matrix multiplication, thereby achieving end-to-end adversarial example optimization without relying on model parameter modifications. Furthermore, this invention manipulates the attention weight distribution between user commands and multimodal inputs during model inference, guiding the model to generate a stable attention shift during the generation phase. Combined with a joint optimization strategy under multiple contextual conditions, this significantly improves the generalization ability of indirect suggestion injection in different user command contexts. Therefore, this invention achieves a general and efficient multimodal indirect suggestion injection method applicable to continuous representation models, discrete representation models, and hybrid architecture models.

[0006] The technical solution of the present invention is as follows:

[0007] This invention discloses a general indirect hint injection method for large multimodal models, comprising the following steps:

[0008] Multimodal data is obtained as benign samples, adversarial perturbations are initialized as Gaussian noise and added to the multimodal data to construct adversarial samples;

[0009] The adversarial sample is input into the modal encoder to obtain the hidden vector sequence. The negative Euclidean distance between each hidden vector and the codebook vector is calculated. After adding Gumbel noise, temperature scaling is performed to obtain the probability distribution. This distribution is then multiplied with the decoder's vocabulary embedding matrix to obtain the multimodal embedding representation.

[0010] Multiple text instructions are encoded to obtain text embedding representations corresponding to text words. Each text embedding representation is concatenated with a multimodal embedding representation and input into a decoder to generate words. The expected cross-entropy between the generated words and the target words is calculated as the adversarial loss.

[0011] For each layer and each attention head of the decoder, the attention weights of the target word to the multimodal word are extracted, summed, and the mean is calculated layer by layer and head by head. The marginal attention loss is also calculated.

[0012] The L2 distance between benign and adversarial samples is calculated as the perceptual concealment loss.

[0013] The adversarial loss, marginal attention loss, and perceptual concealment loss are weighted and summed. The gradient sign at the adversarial sample is calculated by applying gradient backpropagation. The adversarial sample is then updated by gradient descent. The above process is iterated until the maximum number of steps is reached or the target word is successfully generated.

[0014] As a further improvement, the multimodal data described in this invention includes speech data or image data, wherein the speech data is normalized and mapped to intervals. Image data is normalized and mapped to intervals. As an unattacked benign sample, the adversarial perturbation is initialized as random noise with the same dimension as the original multimodal data, and the random noise follows a Gaussian distribution.

[0015] As a further improvement, the codebook described in this invention is a pre-trained codebook for a multimodal large model, used to vectorize continuous multimodal features; the negative Euclidean distance between the latent vector and the codebook vector is used as the logits of discrete word selection, and the Gumbel-Softmax function is used to continuously sample the logits to obtain multimodal embeddings; in the multimodal embedding calculation, discrete one-hot weights are used in the model forward inference stage to maintain the consistency of the original inference behavior of the model, and continuous probability weights are used in the gradient backpropagation stage to achieve end-to-end adversarial optimization.

[0016] As a further improvement, the multiple text instructions described in this invention constitute an auxiliary instruction set to simulate diverse inputs under different user scenarios, and improve the generalization ability of adversarial examples under unknown user instruction conditions by jointly optimizing under multiple different text instruction contexts.

[0017] As a further improvement, the adversarial loss described in this invention is defined as the expected value of the cross-entropy loss between the generated lexical and the target lexical under multiple text instruction contexts.

[0018] As a further improvement, the marginal attention loss described in this invention is used to constrain the attention weights assigned to multimodal lexical units by the model when generating target lexical units to be no less than a preset threshold.

[0019] As a further improvement, the marginal attention loss calculation described in this invention includes:

[0020] During the inference process of the multimodal large model decoder, for each layer and each attention head in the decoder, the attention weights of the target word to the multimodal word are extracted;

[0021] The attention weights are summed in the target word dimension and the multimodal word dimension, and then averaged by layer and attention head to obtain the global multimodal attention weights.

[0022] The global multimodal attention weights are compared with a preset attention threshold, and the marginal attention loss is calculated based on the difference between the two to penalize the model for insufficient attention to multimodal words when generating target words.

[0023] As a further improvement, the perceptual concealment loss described in this invention is obtained by subtracting the benign sample from the adversarial sample element by element and calculating the L2 distance.

[0024] As a further improvement, the weight of the adversarial loss described in this invention is preferably set to 1, and the weights of the marginal attention loss and the perceptual concealment loss are set through ablation experiments.

[0025] The beneficial effects of this invention are as follows:

[0026] This invention discloses a general indirect suggestion injection method for large multimodal models. Addressing the non-differentiability problem arising from the mixed modeling of discrete lexical units and continuous features in multimodal input, this method introduces a sampling-based gradient estimation mechanism. It replaces discrete lexical unit selection and embedding index operations with differentiable probability sampling and matrix multiplication, achieving end-to-end adversarial example optimization. Furthermore, this invention combines multi-context expectation optimization and attention guidance mechanisms, implicitly and explicitly manipulating the model's attention allocation to multimodal data during the inference phase, significantly improving the generalization ability of indirect suggestion injection under unknown user commands and complex context conditions. The method is applicable to continuous representation models, discrete representation models, and large multimodal models with continuous-discrete hybrid architectures. It can stably induce the model to generate target responses in multimodal scenarios such as speech and image processing, thus systematically revealing the security risks of indirect suggestion injection faced by large multimodal models during the inference phase, providing an important technical foundation for related security assessment and protection research.

[0027] This invention addresses the gradient truncation problem caused by multimodal discrete modeling. By reconstructing the discrete word mapping and embedding indexing process into a differentiable form based on probabilistic sampling and matrix multiplication, it achieves unified adversarial optimization for continuous representation models, discrete representation models, and hybrid architecture models while ensuring the model's forward inference process remains unchanged, significantly improving the versatility of indirect suggestion injection methods. To address the context sensitivity issue of adversarial examples, it enhances the cross-context robustness of adversarial examples by introducing multi-instruction context joint optimization. Simultaneously, it further enhances the attention-dominant capability of adversarial examples based on an attention weight manipulation mechanism, effectively improving the stability and generalization ability of adversarial examples under unknown and diverse user instruction conditions. Furthermore, it introduces perceptual concealment constraints during the optimization process, enabling the generated adversarial examples to maintain attack effectiveness while possessing high physical feasibility and perceptual concealment. This invention requires no modification to the model structure or parameters and can be implemented directly during the inference stage, providing an efficient and scalable technical means for assessing the indirect suggestion injection risks faced by large multimodal models in real-world application scenarios. Attached Figure Description

[0028] Figure 1 This is a flowchart of the process steps of the present invention;

[0029] Figure 2 This is a system framework diagram of the present invention. Detailed Implementation

[0030] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are merely intended to assist those skilled in the art in understanding the principles and knowledge of the present invention, and are not intended to limit the scope of the invention, nor should they be considered as limiting the application scenarios of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various modifications or alterations to the invention, but any variations, changes, and transformations made to the embodiments based on the principles and spirit of the invention also fall within the scope defined by the appended claims. And it is obvious that this specification only uses preferred embodiments as examples, and does not need to exhaustively describe all embodiments.

[0031] A general indirect hint injection method for multimodal large models includes the following steps:

[0032] Step S100: Obtain multimodal data as benign samples, initialize adversarial perturbation as Gaussian noise, and add it to the multimodal data to construct adversarial samples;

[0033] Step S200: Input the adversarial sample into the modal encoder to obtain the hidden vector sequence, calculate the negative Euclidean distance between each hidden vector and the codebook vector, add Gumbel noise and perform temperature scaling to obtain the probability distribution, and multiply it with the decoder vocabulary embedding matrix to obtain the multimodal embedding representation.

[0034] Step S300: Encode multiple text instructions to obtain text embedding representations corresponding to text words. Concatenate each text embedding representation with the multimodal embedding representation and input it into the decoder to generate words. Calculate the expected cross-entropy between the generated words and the target words as the adversarial loss.

[0035] Step S400: For each layer and each attention head of the decoder, extract the attention weights of the target word to the multimodal word, sum them up, calculate the average value layer by layer and head by head, and calculate the marginal attention loss.

[0036] Step S500: Calculate the L2 distance between benign samples and adversarial samples as the perceptual concealment loss;

[0037] In step S600, the adversarial loss, marginal attention loss, and perceptual concealment loss are weighted and summed. The gradient sign at the adversarial sample is calculated by applying gradient backpropagation, and the adversarial sample is updated by gradient descent. The above process is iterated until the maximum number of steps is reached or the target word is successfully generated.

[0038] In the above technical solution, in step S100, the multimodal data undergoes normalization processing, wherein the image data is mapped to... In between, voice data is mapped to This serves as a positive example.

[0039] In the above technical solution, step S200 includes:

[0040] Step S210: Input the adversarial examples into the modality encoder of the multimodal large model to encode the data of different modalities and obtain the corresponding latent vector sequences.

[0041] Step S220: Calculate the negative Euclidean distance between each latent vector in the latent vector sequence and the preset codebook vector, and construct a similarity matrix.

[0042] Step S230: Treat the similarity matrix as a probability distribution of discrete words, introduce Gumbel noise, and use the Gumbel-Softmax function for continuous sampling processing.

[0043] Step S240: Multiply the sampled probability distribution matrix with the word embedding matrix of the decoder to obtain a differentiable multimodal embedding representation.

[0044] In the above technical solution, step S300 includes:

[0045] Step S310: The input text instruction is segmented and encoded to obtain the corresponding text tokens, and the text token index decoder vocabulary is used to obtain the text embedding representation.

[0046] Step S320: After concatenating the text embedding representation and the multimodal embedding representation, input the result into the decoder to generate lexical units through an autoregressive method;

[0047] Step S330: Calculate the cross-entropy loss between the predicted word's corresponding logits and the preset target word, and use it as the adversarial loss.

[0048] In the above technical solution, step S400 includes:

[0049] Step S410: For each layer and each attention head in the decoder, extract the attention weights of the target word to the multimodal word, sum the attention weights, and calculate the mean for each layer and attention head.

[0050] Step S420: Calculate the marginal attention loss based on the averaged attention weights to constrain the model's attention distribution to multimodal lexical units.

[0051] In the above technical solution, in step S500, the perceptual concealment loss is obtained by subtracting the benign sample from the adversarial sample element by element and calculating the L2 distance.

[0052] In the above technical solution, in step S600, the weight of the adversarial loss is set to 1, and the weights of the marginal attention loss and the perception concealment loss are set through ablation experiments.

[0053] like Figure 1 As shown, the general indirect hint injection method for multimodal large models proposed in this invention includes:

[0054] Multimodal data is acquired as benign samples, whereby the multimodal data can be speech signals or image data, and then normalized. For the speech modality, the original audio signal is represented as follows: Mapped to interval For image modalities, the original image is represented as: Mapped to interval ;

[0055] When constructing adversarial examples, an adversarial perturbation with the same dimension as the original multimodal data is first initialized. Its initial value is random noise following a Gaussian distribution, and this perturbation is superimposed on the original multimodal data to obtain the initial adversarial example: ;

[0056] like Figure 2 As shown, adversarial examples The input is fed into the modal encoder of a large multimodal model, and continuous multimodal features are extracted through the modal encoder: ;

[0057] A sampling gradient estimation method based on Gumbel-Softmax is adopted to estimate multimodal features. With codebook vector The negative Euclidean distance between them is used as the classification logits: And on this basis, Gumbel noise is introduced. Through temperature parameters After smoothing, a continuous probability distribution is obtained:

[0058]

[0059] Subsequently, this probability distribution is compared with the decoder vocabulary embedding matrix. Perform matrix multiplication to obtain a differentiable multimodal embedding representation: Hard one-hot weights are used in the forward inference stage of the model to maintain consistent inference behavior, while soft probability weights are used in the gradient backpropagation stage to achieve gradient propagation, thereby achieving end-to-end adversarial example optimization.

[0060] To improve context generalization capabilities, a set of auxiliary text instructions is constructed. It is used to simulate the diverse commands that real users may input in different scenarios;

[0061] adversarial examples Each auxiliary instruction is input into the multimodal large model. During the autoregressive decoding process, the model generates the target response text and calculates the generated lexical units and the preset target lexical units. The cross-entropy loss between them. The adversarial loss is defined as:

[0062]

[0063] By jointly optimizing under multiple different contexts, the adversarial examples gradually reduce their dependence on specific text instructions during training, thereby improving their generalization ability under unknown contexts.

[0064] During the model's forward inference process, the decoder calculates the causal attention weight matrix between the target word and historical words: ,in Indicates the first Layer Among the attention heads, the query terms Keyword Attention weights.

[0065] Let the set of terms corresponding to the generated target response text be denoted as: Let the set of lexical units corresponding to the multimodal data be denoted as: Therefore, in each layer and each attention head, the average attention weight obtained from the adversarial example is defined as:

[0066]

[0067] Furthermore, the average value is calculated across all layers and attention heads to obtain the global attention weights:

[0068]

[0069] To prevent adversarial examples from being obscured by user commands in long contexts, this embodiment introduces marginal attention loss:

[0070]

[0071] in A preset attention threshold is used to force the model to allocate sufficient attention to adversarial examples when generating target lexical units.

[0072] To ensure the feasibility and perceptual concealment of adversarial examples in physical space, the L2 distance between the adversarial example and the original example is calculated as the perceptual concealment loss:

[0073]

[0074] The final optimization objective function is defined as:

[0075]

[0076] Based on the above loss function, perform iterative gradient updates on the adversarial examples:

[0077]

[0078] Repeat the above process until the maximum number of iterations is reached or the multimodal large model is successfully induced to stably generate the target response under different context conditions, thereby completing the general indirect hint injection attack.

[0079] The present invention also proposes a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor, when executing the computer program, implements the steps of the above-described general indirect hint injection method for multimodal large models.

[0080] The present invention also proposes a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described general indirect hint injection method for multimodal large models.

[0081] The present invention also proposes a computer program product, which includes a computer program or instructions, and when the computer program or instructions are executed by a processor, implements the steps of the above-described general indirect hint injection method for multimodal large models.

[0082] Finally, it should be noted that the above embodiments and descriptions are only used to illustrate the technical solutions of the present invention and not to limit it. Those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the disclosure of the technical solutions of the present invention, and all such modifications and substitutions should be covered within the protection scope of the claims of the present invention.

Claims

1. A general indirect hint injection method for multimodal large models, characterized in that, Includes the following steps: Multimodal data is obtained as benign samples, adversarial perturbations are initialized as Gaussian noise and added to the multimodal data to construct adversarial samples; The adversarial sample is input into the modal encoder to obtain the hidden vector sequence. The negative Euclidean distance between each hidden vector and the codebook vector is calculated. After adding Gumbel noise, temperature scaling is performed to obtain the probability distribution. This distribution is then multiplied with the decoder's vocabulary embedding matrix to obtain the multimodal embedding representation. Multiple text instructions are encoded to obtain text embedding representations corresponding to text words. Each text embedding representation is concatenated with a multimodal embedding representation and input into a decoder to generate words. The expected cross-entropy between the generated words and the target words is calculated as the adversarial loss. For each layer and each attention head of the decoder, the attention weights of the target word to the multimodal word are extracted, summed, and the mean is calculated layer by layer and head by head. The marginal attention loss is also calculated. The L2 distance between benign and adversarial samples is calculated as the perceptual concealment loss. The adversarial loss, marginal attention loss, and perceptual concealment loss are weighted and summed. The gradient sign at the adversarial sample is calculated by applying gradient backpropagation. The adversarial sample is then updated by gradient descent. The above process is iterated until the maximum number of steps is reached or the target word is successfully generated.

2. The general indirect hint injection method for multimodal large models according to claim 1, characterized in that, The multimodal data includes speech data or image data, wherein the speech data is normalized and mapped to intervals. Image data is normalized and mapped to intervals. As an unattacked benign sample, the initial adversarial perturbation is random noise with the same dimension as the original multimodal data, and the random noise follows a Gaussian distribution.

3. The general indirect hint injection method for multimodal large models according to claim 1 or 2, characterized in that, The codebook is a pre-trained codebook for a multimodal large model, used to vectorize continuous multimodal features; The negative Euclidean distance between the latent vector and the codebook vector is used as the logits of discrete token selection, and the Gumbel-Softmax function is used to continuously sample the logits to obtain multimodal embeddings. The multimodal embedding calculation uses discrete one-hot weights in the forward inference stage of the model to maintain the consistency of the original inference behavior of the model, and uses continuous probability weights in the gradient backpropagation stage to achieve end-to-end adversarial optimization.

4. The general indirect hint injection method for multimodal large models according to claim 3, characterized in that, The multiple text instructions constitute an auxiliary instruction set, which is used to simulate diverse inputs under different user scenarios. By jointly optimizing under multiple different text instruction contexts, the generalization ability of adversarial examples under unknown user instruction conditions can be improved.

5. The general indirect hint injection method for multimodal large models according to claim 4, characterized in that, The adversarial loss is defined as the expected value of the cross-entropy loss between the generated lexical and the target lexical under multiple text instruction contexts.

6. The general indirect hint injection method for multimodal large models according to claim 1, 2, 4, or 5, characterized in that, The marginal attention loss is used to constrain the model to assign attention weights to multimodal lexical units when generating target lexical units to be no less than a preset threshold.

7. The general indirect hint injection method for multimodal large models according to claim 6, characterized in that, The calculation of marginal attention loss includes: During the inference process of the multimodal large model decoder, for each layer and each attention head in the decoder, the attention weights of the target word to the multimodal word are extracted; The attention weights are summed in the target lexical dimension and the multimodal lexical dimension, and then averaged by layer and attention head to obtain the global multimodal attention weights. The global multimodal attention weights are compared with a preset attention threshold, and the marginal attention loss is calculated based on the difference between the two to penalize the model for insufficient attention to multimodal words when generating target words.

8. The general indirect hint injection method for multimodal large models according to claim 7, characterized in that, The perceived concealment loss is obtained by subtracting the benign sample from the adversarial sample element by element and calculating the L2 distance.

9. The general indirect hint injection method for multimodal large models according to claim 1, 2, 4, 5, 7, or 8, characterized in that, The weight of the adversarial loss is preferably set to 1, and the weights of the marginal attention loss and the perceptual concealment loss are set through ablation experiments.

Images