Method and device for generating multimodal large language model adversarial sample based on cooperation of image disturbance and text guidance

Abstract

The invention provides a multi-mode large language model adversarial sample generation method and device based on cooperation of image disturbance and text guidance, and relates to the technical field of computers.The method comprises the steps that disturbance is added to a target image through a malicious question set and a guidance prompt word of the current round to generate an adversarial image, inputting the confrontation image, the malicious question set and the guide word of the current round into the model to obtain reply content for each malicious question; generating common questions and improvement suggestions aiming at the reply contents of the model based on the reply contents of the model aiming at each malicious question, and generating a guide prompt word of the next round; and iterating the confrontation image and the guide prompt word until a preset end condition is met, and generating a confrontation sample. According to the image disturbance and text guidance collaborative multi-modal large language model adversarial sample generation method and device provided by the invention, the effect of adversarial samples during adversarial sample testing is greatly improved by introducing extra information on vision and text.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular, to a method and device for generating adversarial samples for multimodal large language models by coordinating image perturbation and text guidance. Background Art

[0002] Multimodal Large Language Models (MLLMs) are artificial intelligence systems that combine large language models (LLMs) with multimodal data processing capabilities, aiming to understand and generate cross-modal content (such as text, images, audio, video, etc.).

[0003] In related technologies, using adversarial samples to conduct adversarial sample testing on multimodal large language models can not only evaluate the vulnerability of the model when facing malicious inputs, but also discover potential security defects of the model, providing targeted goals for improving the security of the model.

[0004] However, when the adversarial samples obtained by the adversarial sample generation methods in related technologies are used for adversarial sample testing, the content generated by the model often deviates from the instruction target and the generated content is empty, greatly affecting the quality of the test results. Summary of the Invention

[0005] The purpose of this application is to provide a method and device for generating adversarial samples for multimodal large language models by coordinating image perturbation and text guidance. By introducing additional visual and text information, the effect of adversarial samples in adversarial sample testing is greatly improved.

[0006] This application provides a method for generating adversarial samples for multimodal large language models by coordinating image perturbation and text guidance, including: Using the malicious problems in the malicious problem set and the guiding prompt words of the current round to add perturbations to the target image, generating adversarial images, and inputting the adversarial images, the malicious problems in the malicious problem set, and the guiding words of the current round into a multimodal large language model to obtain the response content of the multimodal large language model for each malicious problem; the malicious problem set contains multiple malicious problems; the target image is: a given image used to generate adversarial images; based on the response content of the multimodal large language model for each malicious problem, generating common problems and improvement suggestions for the response content of the multimodal large language model, and generating the guiding prompt words for the next round based on the common problems and the improvement suggestions; using the guiding prompt words of the next round to iterate the adversarial images and the guiding prompt words until a preset end condition is met, generating adversarial samples; wherein, in each iteration, the perturbations of the target image are updated, new adversarial images are generated, and the response content generated by the multimodal large language model is guided based on the newly generated adversarial images and the newly generated guiding prompt words; the adversarial samples include: guiding prompt words and adversarial images.

[0007] Optionally, the step of using the malicious problems in the malicious problem set and the guiding prompt words of the current round to add perturbations to the target image and generate adversarial images includes: optimizing the generation process of the adversarial images with minimizing cross-entropy as the optimization objective to obtain the adversarial images with perturbations added on the basis of the target image, so as to maximize the probability of the model generating positive responses; wherein, the cross-entropy of the adversarial images is calculated based on the expected output results of the model for each malicious problem, the text input content of the model, and the generated adversarial images; the text input content of the model includes: the guiding prompt words of the current round, the multiple malicious problems, and the expected response prefix of the model; the expected response prefix of the model is used to guide the prefix of the response content of the model; the perturbations of the target image are updated based on the momentum term and are constrained by the maximum perturbation amplitude.

[0008] Optionally, the adversarial images can be optimized by the following formula: Wherein, N is the total number of malicious problems in the malicious problem set, represents the expected output result of the model for the i th malicious problem; I is the target image, is the perturbation, indicates that the perturbation is limited within the maximum perturbation amplitude; is the generated adversarial image; T is the input text of the model.

[0009] Optionally, for the response content of the multimodal large language model for each malicious question, common problems and improvement suggestions for the response content of the multimodal large language model are generated, and the guiding prompt words for the next round are generated based on the common problems and the improvement suggestions, including: using the first agent to score the response content corresponding to each malicious question and output modification suggestions for each response content; using the second agent to obtain common problems and improvement suggestions for the response content of the multimodal large language model based on the score of each response content, the modification suggestions corresponding to each response content, and the guiding prompt words of the current round; using the third agent to generate the guiding prompt words for the next round based on the common problems, the improvement suggestions, and the guiding prompt words of the current round.

[0010] Optionally, the step of using the first agent to score the response content corresponding to each malicious question and output modification suggestions for each response content can be represented by the following formula: where is the i-th harmful question, is the response content of the model for the i-th harmful question; is the score of the response content of the first agent for the i-th harmful question, is the modification suggestion generated by the first agent for the i-th harmful question and the corresponding response content.

[0011] Optionally, the step of using the second agent to obtain common problems and improvement suggestions for the response content of the multimodal large language model based on the score of each response content, the modification suggestions corresponding to each response content, and the guiding prompt words of the current round can be represented by the following formula: where M represents the common problem, U is the improvement suggestion; is the guiding prompt word of the previous round.

[0012] Optionally, the step of using the third agent to generate the guiding prompt words for the next round based on the common problems, the improvement suggestions, and the guiding prompt words of the current round can be represented by the following formula: where is the k guiding prompt word generated for the target at the

[0013] The present application also provides a multi-modal large language model adversarial sample generation device that collaborates image perturbation and text guidance, including: An image perturbation module, configured to add perturbations to a target image by using malicious questions in a malicious question set and guiding prompt words of the current round to generate an adversarial image, and input the adversarial image, the malicious questions in the malicious question set, and the guiding words of the current round into the multi-modal large language model to obtain the response content of the multi-modal large language model for each malicious question; the malicious question set contains multiple malicious questions; the target image is a given image used to generate the adversarial image; a text guidance module, configured to generate common questions and improvement suggestions for the response content of the multi-modal large language model based on the response content of the multi-modal large language model for each malicious question, and generate guiding prompt words for the next round based on the common questions and the improvement suggestions; a sample generation module, configured to perform iteration of the adversarial image and the guiding prompt words by using the guiding prompt words of the next round, and generate adversarial samples after meeting a preset end condition; wherein, in each round of iteration, the perturbation of the target image is updated, a new adversarial image is generated, and the response content generated by the multi-modal large language model is guided by using the newly generated guiding prompt words based on the newly generated adversarial image; the adversarial samples include: guiding prompt words and adversarial images.

[0014] Optionally, the image perturbation module is specifically configured to optimize the generation process of the adversarial image with minimizing cross-entropy as the optimization objective to obtain an adversarial image with perturbations added on the basis of the target image, so as to maximize the probability of the model generating a positive response; wherein, the cross-entropy of the adversarial image is calculated based on the expected output result of the model for each malicious question, the text input content of the model, and the generated adversarial image; the text input content of the model includes: the guiding prompt words of the current round, the multiple malicious questions, and the expected response prefix of the model; the expected response prefix of the model is used to guide the prefix of the response content of the model; the perturbation of the target image is updated based on the momentum term and is constrained by the maximum perturbation amplitude.

[0015] Optionally, the adversarial image can be optimized by the following formula: wherein, N is the total number of malicious questions in the malicious question set, represents the expected output result of the model for the i th malicious question; I is the target image, is the perturbation, means that the perturbation is restricted within the maximum perturbation amplitude; is the generated adversarial image;T The input text for the model.

[0016] Optionally, the text guidance module is specifically configured to use a first intelligent agent to score the response content corresponding to each malicious question and output modification suggestions for each response content; the text guidance module is further specifically configured to use a second intelligent agent to obtain common problems and improvement suggestions for the response content of the multi-modal large language model based on the score of each response content, the modification suggestions corresponding to each response content, and the guidance prompt words of the current round; the text guidance module is further specifically configured to use a third intelligent agent to generate the guidance prompt words for the next round based on the common problems, the improvement suggestions, and the guidance prompt words of the current round.

[0017] Optionally, the step of the text guidance module using a first intelligent agent to score the response content corresponding to each malicious question and output modification suggestions for each response content can be represented by the following formula: Where, is the i-th harmful question, is the response content of the model for the i-th harmful question; is the score of the response content of the first intelligent agent for the i-th harmful question, is the modification suggestion generated by the first intelligent agent for the i-th harmful question and the corresponding response content.

[0018] Optionally, the step of the text guidance module using a second intelligent agent to obtain common problems and improvement suggestions for the response content of the multi-modal large language model based on the score of each response content, the modification suggestions corresponding to each response content, and the guidance prompt words of the current round can be represented by the following formula: Where, M represents the common problem, U is the improvement suggestion; is the guidance prompt word of the previous round.

[0019] Optionally, the step of the text guidance module using a third intelligent agent to generate the guidance prompt words for the next round based on the common problems, the improvement suggestions, and the guidance prompt words of the current round can be represented by the following formula: Where, is for the k round of iteration for the target generated guidance prompt word.

[0020] The present application also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the multi-modal large language model adversarial sample generation method for image perturbation and text guidance collaboration as described in any one of the above.

[0021] The present application also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the steps of the multi-modal large language model adversarial sample generation method for image perturbation and text guidance collaboration as described in any one of the above.

[0022] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the multi-modal large language model adversarial sample generation method for image perturbation and text guidance collaboration as described in any one of the above.

[0023] For the multi-modal large language model adversarial sample generation method and device for image perturbation and text guidance collaboration provided by the present application, first, use the malicious questions in the malicious question set and the guidance prompt words of the current round to add perturbations to the target image to generate an adversarial image, and input the adversarial image, the malicious questions in the malicious question set, and the guidance words of the current round into the multi-modal large language model to obtain the response content of the multi-modal large language model for each malicious question; the malicious question set contains multiple malicious questions; the target image is: a given image used to generate an adversarial image; then, based on the response content of the multi-modal large language model for each malicious question, generate common questions and improvement suggestions for the response content of the multi-modal large language model, and generate the guidance prompt words for the next round based on the common questions and the improvement suggestions; finally, use the guidance prompt words for the next round to iterate the adversarial image and the guidance prompt words until a preset end condition is met, and then generate adversarial samples; where, in each iteration, the perturbation of the target image is updated, a new adversarial image is generated, and the response content generated by the multi-modal large language model is guided using the newly generated guidance prompt words based on the newly generated adversarial image; the adversarial samples include: guidance prompt words and adversarial images. In this way, by introducing additional visual and text information, the effect of adversarial samples in adversarial sample testing is significantly improved. Description of the Drawings

[0024] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0025] Figure 1 It is a schematic structural diagram of the framework of the multi-modal large language model adversarial sample generation method that combines image perturbation and text guidance provided by this application; Figure 2 It is a schematic flowchart of the multi-modal large language model adversarial sample generation method that combines image perturbation and text guidance provided by this application; Figure 3 It is a schematic structural diagram of the multi-modal large language model adversarial sample generation device that combines image perturbation and text guidance provided by this application; Figure 4 It is a schematic structural diagram of the electronic device provided by this application. Detailed implementation manners

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below with reference to the accompanying drawings in this application. Obviously, the described embodiments are some, but not all, of the embodiments of this application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without creative efforts shall fall within the protection scope of this application.

[0027] The terms "first", "second", etc. in the description and claims of this application are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of this application can be implemented in an order different from those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same category, and the number of objects is not limited. For example, the first object can be one or multiple. In addition, "and / or" in the description and claims means at least one of the connected objects, and the character " / " generally indicates an "or" relationship between the associated objects before and after.

[0028] In the related art, some key problems still exist in the adversarial examples generated by the adversarial example generation method during testing, resulting in poor quality of the results obtained during testing. Specifically, there are two main problems in the examples with low quality of the generation method in the prior art: one is that the generated content deviates from the instruction target, that is, the response generated by the model during the adversarial example test fails to conform to the original malicious problem; the other is that the generated content is empty, that is, the response generated by the model is superficially related to the preset request, but lacks actual harmfulness in content. Therefore, the technical problem of the present invention is how to overcome the above technical problems and improve the effectiveness of adversarial example generation, especially in multi-modal large language models, by introducing additional visual and textual information to enhance the success rate of adversarial example testing. Based on this, the embodiments of the present application provide a method for generating adversarial examples for multi-modal large language models that combines image perturbation and text guidance. This method uses a combination of image perturbation and text guidance, and through the collaborative optimization of the two modalities, can greatly improve the effect of generating adversarial examples.

[0029] This method simultaneously utilizes the unique roles of images and text, collaboratively optimizes these two modalities, thereby greatly enhancing the effect of adversarial example testing. Specifically, this method searches for image perturbations through momentum-based projected gradient descent (momentum-based PGD) to bypass the security mechanism of the MLLM, and at the same time iteratively guides the text through a multi-agent system to ensure that the model generates more high-quality specific content that conforms to the malicious query intention. Through the iterative optimization and collaboration of the image and text modalities, significant adversarial example testing effects have been achieved on multiple standard data sets when conducting adversarial sample tests on multiple multi-modal large language models.

[0030] As Figure 1 shown, it is a schematic diagram of the framework structure of the method for generating adversarial examples for multi-modal large language models that combines image perturbation and text guidance provided by the embodiments of the present application. This method mainly includes two core steps: image perturbation and text guidance. 1. Image perturbation: First, the input image and the malicious problem set are used to generate an adversarial image. This adversarial image is optimized through the momentum-based projected gradient descent attack (momentum-based PGD) method, aiming to bypass the security mechanism of the target model. 2. Text guidance: After generating the adversarial image, the generated image is sent to the text guidance stage, and the text prompt is optimized through the multi-agent method to ensure that the generated response not only conforms to the intention of the malicious query but also avoids the problem of insufficient harmfulness. 3. Iterative optimization: After each update of the text guidance, the process returns to the image perturbation stage, and continues to optimize the perturbation in combination with the new text prompt. This process is repeated until a predetermined number of iterations is reached, or the generated adversarial example can meet the test requirements.

[0031] The following will combine the accompanying drawings and, through specific embodiments and their application scenarios, elaborate in detail on the method for generating adversarial samples of a multimodal large language model that synergizes image perturbation and text guidance provided by the embodiments of the present application.

[0032] As Figure 2 shown, a method for generating adversarial samples of a multimodal large language model that synergizes image perturbation and text guidance provided by the embodiments of the present application may include the following steps 201 to 203: Step 201: Use the malicious questions in the malicious question set and the guiding prompt words of the current round to add perturbations to the target image, generate an adversarial image, and input the adversarial image, the malicious questions in the malicious question set, and the guiding words of the current round into the multimodal large language model to obtain the response content of the multimodal large language model for each malicious question.

[0033] Among them, the malicious question set contains multiple malicious questions; the target image is: a given image used to generate an adversarial image.

[0034] Exemplarily, based on the given target image, the embodiments of the present application use the target image and the malicious question set to generate an adversarial image, and this adversarial image is optimized by the momentum-based projected gradient descent attack method, aiming to bypass the security mechanism of the target model.

[0035] Specifically, in the above step 201, the step of generating an adversarial image may further include the following step 201a: Step 201a: Optimize the generation process of the adversarial image with minimizing cross-entropy as the optimization objective to obtain an adversarial image with perturbations added on the basis of the target image, so as to maximize the probability that the model generates a positive response.

[0036] Among them, the cross-entropy of the adversarial image is calculated based on the expected output results of the model for each malicious question, the text input content of the model, and the generated adversarial image; the text input content of the model includes: the guiding prompt words of the current round, the multiple malicious questions, and the expected reply prefix of the model; the expected reply prefix of the model is used to guide the prefix of the reply content of the model; the perturbation of the target image is updated based on the momentum term and is constrained by the maximum perturbation amplitude.

[0037] Exemplarily, considering the continuity of the image and the extensive search space, in the embodiments of the present application, the security mechanism of the multimodal large language model MLLM is bypassed by searching for adversarial perturbations. That is, first, the given target image I and the guiding prompt words (it should be noted that the guiding prompt words are null values when generating the adversarial image for the first time) are provided. Then, an adversarial image is generated through the momentum-based projected gradient descent attack method The adversarial image can maximize the probability that the multi-modal large language model MLLM generates a positive response (i.e., enables the model to respond positively to questions), which can be represented by the following formula (1): (Formula 1) Wherein, N is the total number of malicious questions in the malicious question set, represents the expected output result of the model for the i th malicious question; I is the target image, is the perturbation, indicating that the perturbation is restricted within the maximum perturbation amplitude; is the generated adversarial image; T is the input text of the model.

[0038] Exemplarily, to assist the optimization process, in the embodiments of the present application, a prefix G of the expected model response is added after the malicious question as a guide to ensure that the multi-modal large language model can generate a response consistent with the adversarial sample generation target, that is, to guide the model to start the response with the prefix G, which can be specifically represented by the following formula (2): (Formula 2) Wherein, is the guiding prompt word, , is the beginning of the model response content.

[0039] Exemplarily, to address the instability in the optimization process, a momentum term is added to accelerate convergence and stabilize the perturbation search process. The momentum term can be represented by the following formula (3): (Formula 3) Wherein, is the momentum after the t-th iteration, is the momentum coefficient, is at the th step of the image gradient. Based on this, the perturbation can be updated by the following formula (4): (Formula 4) Wherein, represents the perturbation at the th step, and this perturbation is updated through the learning rate and the momentum . represents the original picture, The amplitude of the disturbance is limited, and the clamp function ensures that the disturbance remains within [ Within the range, and This ensures that the final adversarial image is within the range that the image can represent.

[0040] Step 202: Based on the response content of the multimodal large language model to each malicious question, generate common problems and improvement suggestions for the response content of the multimodal large language model, and generate guiding prompt words for the next round based on the common problems and the improvement suggestions.

[0041] For example, after adding perturbations to the target image, the entire dataset will enter the text guidance stage, in which the text prompts are optimized to ensure that the generated responses not only meet the intention of the malicious query but also contain sufficient harmfulness.

[0042] Specifically, the above step 202 may further include the following steps 202a1 to 202a3: Step 202a1: Use the first agent to score the reply content corresponding to each malicious question and output modification suggestions for each reply content.

[0043] Step 202a2: Utilize the second agent to obtain common problems and improvement suggestions for the reply content of the multimodal large language model based on the score of each reply content, the modification suggestions corresponding to each reply content, and the guiding prompt words of the current round.

[0044] Step 202a3: Utilize a third agent to generate guidance prompt words for the next round based on the common problems, the improvement suggestions, and the guidance prompt words for the current round.

[0045] For example, Figure 1 As shown, the embodiment of the present application adopts a multi-agent method, which includes an "evaluator" agent, a "summarizer" agent, and a "reviser" agent, namely the first agent, the second agent, and the third agent mentioned above respectively.

[0046] 1. Evaluator Agent: The evaluator is responsible for assigning a score between 1 and 10 based on each malicious question and the corresponding answer from the MLLM (for instruction compliance, 10 indicates complete compliance; for harmfulness, 10 indicates explicit harmful details), and providing modification suggestions, which can be expressed as the following formula 5: (Formula 5) in, is the i-th harmful problem, is the response content of the model for the i-th harmful problem; is the score of the response content of the first agent for the i-th harmful problem, is the modification suggestion generated by the first agent for the i-th harmful problem and the corresponding response content.

[0047] 2. Summarizer agent: The summarizer is responsible for aggregating the scores and suggestions of the evaluator, combining the original question and answer, and the guiding prompt words obtained in the previous round, finding common problems in the current answer, and providing improvement suggestions, which can be represented by the following formula six: (Formula six) where, M represents the common problem, U is the improvement suggestion; is the guiding prompt word of the previous round.

[0048] 3. Revisor agent: The revisor is responsible for generating the final guiding prompt word according to the common problem M and improvement suggestion U provided by the summarizer and the guiding prompt word of the previous round, which can be represented by the following formula seven: (Formula seven) where, is the k guiding prompt word generated for the target at the

[0049] It should be noted that the entire process will generate guiding prompt words from the two perspectives of "instruction following" and "harmfulness" respectively, and these two guiding prompt words will be concatenated as the final guiding prompt word obtained in the text guiding stage.

[0050] Step 203: Use the guiding prompt word of the next round to iterate the adversarial image and the guiding prompt word until the preset end condition is met, and then generate an adversarial sample.

[0051] Among them, in each round of iteration, the perturbation of the target image will be updated, a new adversarial image will be generated, and based on the newly generated adversarial image, the guiding prompt word will be used to guide the response content generated by the multi-modal large language model; the adversarial sample includes: the guiding prompt word and the adversarial image.

[0052] Exemplarily, after obtaining the guiding prompt for the next round, it is necessary to repeatedly execute the above steps 201 and 202 using this guiding prompt, update the perturbation of the target image using the above formula four, and input the obtained adversarial image into the multi-modal large language model to obtain new response content. Then, generate a new guiding prompt based on the new response content, and so on, until after meeting the preset end condition, the final guiding prompt and the adversarial image obtained are used as adversarial samples.

[0053] The method for generating adversarial samples for a multi-modal large language model that collaborates image perturbation and text guidance provided by the embodiments of the present application. First, use the malicious questions in the malicious question set and the guiding prompt for the current round to add perturbations to the target image to generate an adversarial image, and input the adversarial image, the malicious questions in the malicious question set, and the guiding word for the current round into the multi-modal large language model to obtain the response content of the multi-modal large language model for each malicious question; the malicious question set contains multiple malicious questions; the target image is: a given image used to generate the adversarial image; then, based on the response content of the multi-modal large language model for each malicious question, generate common questions and improvement suggestions for the response content of the multi-modal large language model, and generate the guiding prompt for the next round based on the common questions and the improvement suggestions; finally, use the guiding prompt for the next round to perform iteration of the adversarial image and the guiding prompt until after meeting the preset end condition, generate adversarial samples; where, in each iteration, the perturbation of the target image will be updated, and a new adversarial image will be generated, and based on the newly generated adversarial image, use the newly generated guiding prompt to guide the response content generated by the multi-modal large language model; the adversarial samples include: guiding prompts and adversarial images. Thus, by introducing additional visual and text information, the effect of the adversarial samples during the adversarial sample test is significantly improved.

[0054] It should be noted that for the method for generating adversarial samples for a multi-modal large language model that collaborates image perturbation and text guidance provided by the embodiments of the present application, the execution subject can be a device for generating adversarial samples for a multi-modal large language model that collaborates image perturbation and text guidance, or a control module in the device for generating adversarial samples for a multi-modal large language model that collaborates image perturbation and text guidance and is used to execute the method for generating adversarial samples for a multi-modal large language model that collaborates image perturbation and text guidance. In the embodiments of the present application, taking the device for generating adversarial samples for a multi-modal large language model that collaborates image perturbation and text guidance to execute the method for generating adversarial samples for a multi-modal large language model that collaborates image perturbation and text guidance as an example, the device for generating adversarial samples for a multi-modal large language model that collaborates image perturbation and text guidance provided by the embodiments of the present application is described.

[0055] It should be noted that in the embodiments of the present application, the multi-modal large language model adversarial sample generation method combining image perturbation and text guidance shown in the above-mentioned respective method drawings is exemplarily described by taking one drawing in the embodiments of the present application as an example. In specific implementation, the multi-modal large language model adversarial sample generation method combining image perturbation and text guidance shown in the above-mentioned respective method drawings can also be implemented in combination with any other combinable drawings schemed in the above-mentioned embodiments, which will not be elaborated here.

[0056] The multi-modal large language model adversarial sample generation device provided by the present application will be described below. The following description can be mutually referred to corresponding to the multi-modal large language model adversarial sample generation method described above.

[0057] Figure 3 It is a structural schematic diagram of the multi-modal large language model adversarial sample generation device provided by the embodiments of the present application. As Figure 3 shown, it specifically includes: An image perturbation module 301, configured to add perturbations to a target image by using malicious questions in a malicious question set and guiding prompt words of the current round to generate an adversarial image, and input the adversarial image, the malicious questions in the malicious question set, and the guiding words of the current round into a multi-modal large language model to obtain reply content of the multi-modal large language model for each malicious question; the malicious question set contains multiple malicious questions; the target image is a given image for generating an adversarial image; a text guidance module 302, configured to generate common questions and improvement suggestions for the reply content of the multi-modal large language model based on the reply content of the multi-modal large language model for each malicious question, and generate guiding prompt words for the next round based on the common questions and the improvement suggestions; a sample generation module 303, configured to perform iteration of the adversarial image and the guiding prompt words by using the guiding prompt words for the next round until a preset end condition is met, and then generate an adversarial sample; wherein, in each round of iteration, the perturbation of the target image is updated, a new adversarial image is generated, and the reply content generated by the multi-modal large language model is guided by using the newly generated guiding prompt words based on the newly generated adversarial image; the adversarial sample includes: guiding prompt words and an adversarial image.

[0058] Optionally, the image perturbation module 301 is specifically configured to optimize the generation process of the adversarial image with minimizing cross-entropy as the optimization objective, obtain the adversarial image with perturbations added on the basis of the target image, so as to maximize the probability that the model generates a positive response; wherein, the cross-entropy of the adversarial image is calculated based on the expected output result of the model for each malicious question, the text input content of the model, and the generated adversarial image; the text input content of the model includes: the guiding prompt word of the current round, the plurality of malicious questions, and the expected reply prefix of the model; the expected reply prefix of the model is used to guide the prefix of the reply content of the model; the perturbation of the target image is updated based on the momentum term and is constrained by the maximum perturbation amplitude.

[0059] Optionally, the adversarial image can be optimized by the following formula: Wherein, N is the total number of malicious questions in the malicious question set, represents the expected output result of the model for the i th malicious question; I is the target image, is the perturbation, means that the perturbation is restricted within the maximum perturbation amplitude; is the generated adversarial image; T is the input text of the model.

[0060] Optionally, the text guidance module 302 is specifically configured to use the first intelligent agent to score the reply content corresponding to each malicious question and output modification suggestions for each reply content; the text guidance module 302 is specifically further configured to use the second intelligent agent to obtain the common problems and improvement suggestions for the reply content of the multimodal large language model based on the score of each reply content, the modification suggestions corresponding to each reply content, and the guiding prompt word of the current round; the text guidance module 302 is specifically further configured to use the third intelligent agent to generate the guiding prompt word for the next round based on the common problems, the improvement suggestions, and the guiding prompt word of the current round.

[0061] Optionally, the step that the text guidance module 302 uses the first intelligent agent to score the reply content corresponding to each malicious question and output modification suggestions for each reply content can be represented by the following formula: Wherein, is the i-th harmful question, is the reply content of the model for the i-th harmful question; The score for the response content of the first intelligent agent for the i-th harmful problem, The modification suggestions generated by the first intelligent agent for the i-th harmful problem and the corresponding response content.

[0062] Optionally, the step of the text guidance module 302 using the second intelligent agent to obtain the common problems and improvement suggestions for the response content of the multimodal large language model based on the score of each response content, the modification suggestions corresponding to each response content, and the guidance prompt words of the current round can be represented by the following formula: where, M represents the common problems, U is the improvement suggestion; is the guidance prompt word of the previous round.

[0063] Optionally, the step of the text guidance module 302 using the third intelligent agent to generate the guidance prompt word of the next round based on the common problems, the improvement suggestions, and the guidance prompt word of the current round can be represented by the following formula: where, is the k guidance prompt word generated for the target at the iteration of the

[0064] The device for generating adversarial samples of a multimodal large language model that combines image perturbation and text guidance provided by this application first adds perturbations to the target image using the malicious questions in the malicious question set and the guidance prompt words of the current round to generate an adversarial image, and inputs the adversarial image, the malicious questions in the malicious question set, and the guidance words of the current round into the multimodal large language model to obtain the response content of the multimodal large language model for each malicious question; the malicious question set contains multiple malicious questions; the target image is the given image used to generate the adversarial image; then, based on the response content of the multimodal large language model for each malicious question, common problems and improvement suggestions for the response content of the multimodal large language model are generated, and the guidance prompt words for the next round are generated based on the common problems and the improvement suggestions; finally, the adversarial image and the guidance prompt words are iterated using the guidance prompt words for the next round until the preset end condition is met, and then adversarial samples are generated; where, in each iteration, the perturbation of the target image is updated, a new adversarial image is generated, and the response content generated by the multimodal large language model is guided using the newly generated guidance prompt words based on the newly generated adversarial image; the adversarial samples include: guidance prompt words and adversarial images. In this way, by introducing additional visual and text information, the effect of the adversarial samples during the adversarial sample test is significantly improved.

[0065] Figure 4 An entity structure diagram of an electronic device is exemplified, as Figure 4As shown in the figure, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communication bus 440. Among them, the processor 410, the communications interface 420, and the memory 430 complete communication with each other through the communication bus 440. The processor 410 may call the logical instructions in the memory 430 to execute a method for generating adversarial samples for a multimodal large language model that collaborates image perturbation and text guidance. The method includes: First, use the malicious questions in the malicious question set and the guidance prompt words of the current round to add perturbations to the target image to generate an adversarial image, and input the adversarial image, the malicious questions in the malicious question set, and the guidance words of the current round into the multimodal large language model to obtain the response content of the multimodal large language model for each malicious question; the malicious question set contains multiple malicious questions; the target image is: a given image used to generate an adversarial image; After that, based on the response content of the multimodal large language model for each malicious question, generate common questions and improvement suggestions for the response content of the multimodal large language model, and generate the guidance prompt words for the next round based on the common questions and the improvement suggestions; Finally, use the guidance prompt words for the next round to iterate the adversarial image and the guidance prompt words until a preset end condition is met, and then generate adversarial samples; where, in each iteration, the perturbation of the target image is updated, and a new adversarial image is generated, and based on the newly generated adversarial image, the response content generated by the multimodal large language model is guided using the newly generated guidance prompt words; the adversarial samples include: guidance prompt words and adversarial images. In this way, by introducing additional visual and text information, the effect of adversarial samples in adversarial sample testing is greatly improved.

[0066] In addition, when the logical instructions in the above-mentioned memory 430 can be implemented in the form of software functional units and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0067] On the other hand, the present application also provides a computer program product. The computer program product includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the multi-modal large language model adversarial sample generation method that combines image perturbation and text guidance provided by the above-mentioned various methods. The method includes: First, use the malicious questions in the malicious question set and the guidance prompt words of the current round to add perturbations to the target image to generate an adversarial image, and input the adversarial image, the malicious questions in the malicious question set, and the guidance words of the current round into the multi-modal large language model to obtain the response content of the multi-modal large language model for each malicious question; the malicious question set contains multiple malicious questions; the target image is: a given image used to generate an adversarial image; After that, based on the response content of the multi-modal large language model for each malicious question, generate common questions and improvement suggestions for the response content of the multi-modal large language model, and generate the guidance prompt words for the next round based on the common questions and the improvement suggestions; Finally, use the guidance prompt words of the next round to iterate the adversarial image and the guidance prompt words until a preset end condition is met, and then generate adversarial samples; wherein, in each iteration, the perturbation of the target image is updated, a new adversarial image is generated, and the response content generated by the multi-modal large language model is guided based on the newly generated adversarial image and the newly generated guidance prompt words; the adversarial samples include: guidance prompt words and adversarial images. In this way, by introducing additional visual and text information, the effect of the adversarial samples in the adversarial sample test is greatly improved.

[0068] In another aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is used to implement the multi-modal large language model adversarial sample generation method that coordinates image perturbation and text guidance provided above. The method includes: First, use the malicious questions in the malicious question set and the guidance prompt words of the current round to add perturbations to the target image to generate an adversarial image, and input the adversarial image, the malicious questions in the malicious question set, and the guidance words of the current round into the multi-modal large language model to obtain the response content of the multi-modal large language model for each malicious question; the malicious question set contains multiple malicious questions; the target image is: a given image used to generate an adversarial image; After that, based on the response content of the multi-modal large language model for each malicious question, generate common questions and improvement suggestions for the response content of the multi-modal large language model, and generate the guidance prompt words for the next round based on the common questions and the improvement suggestions; Finally, use the guidance prompt words of the next round to iterate the adversarial image and the guidance prompt words until the preset end condition is met, and then generate adversarial samples; where, in each iteration, the perturbation of the target image is updated, and a new adversarial image is generated, and based on the newly generated adversarial image, the response content generated by the multi-modal large language model is guided using the newly generated guidance prompt words; the adversarial samples include: guidance prompt words and adversarial images. In this way, by introducing additional visual and text information, the effect of adversarial samples during adversarial sample testing is greatly improved.

[0069] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0070] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, also by hardware. Based on such an understanding, the above technical solutions, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present application.

Claims

1. A method for generating adversarial samples of a multimodal large language model that combines image perturbation and text guidance, characterized in that, Including: Using the malicious problems in the malicious problem set and the guiding prompt words of the current round to add perturbations to the target image, generating adversarial images, and inputting the adversarial images, the malicious problems in the malicious problem set, and the guiding words of the current round into the multimodal large language model to obtain the response content of the multimodal large language model for each malicious problem; the malicious problem set contains multiple malicious problems; the target image is the given image used to generate the adversarial image; Based on the response content of the multimodal large language model for each malicious problem, generating common problems and improvement suggestions for the response content of the multimodal large language model, and generating the guiding prompt words for the next round based on the common problems and the improvement suggestions; Using the guiding prompt words of the next round to iterate the adversarial images and the guiding prompt words until the preset end condition is met, and generating adversarial samples; Among them, in each iteration, the perturbation of the target image will be updated, a new adversarial image will be generated, and the response content generated by the multimodal large language model will be guided based on the newly generated adversarial image and the newly generated guiding prompt words; the adversarial samples include: guiding prompt words and adversarial images.

2. The method according to claim 1, wherein The step of using the malicious problems in the malicious problem set and the guiding prompt words of the current round to add perturbations to the target image and generate adversarial images includes: Optimizing the generation process of the adversarial image with minimizing cross-entropy as the optimization objective to obtain the adversarial image with perturbations added on the basis of the target image, so as to maximize the probability of the model generating positive responses; Among them, the cross-entropy of the adversarial image is calculated based on the expected output results of the model for each malicious problem, the text input content of the model, and the generated adversarial image; the text input content of the model includes: the guiding prompt words of the current round, the multiple malicious problems, and the expected response prefix of the model; the expected response prefix of the model is used to guide the prefix of the response content of the model; the perturbation of the target image is updated based on the momentum term and is constrained by the maximum perturbation amplitude.

3. The method according to claim 2, characterized in that, The adversarial image can be optimized by the following formula: Among them, N is the total number of malicious problems in the malicious problem set, represents the expected output result of the model for the i th malicious problem; I is the target image, is the perturbation, indicating that the perturbation is restricted within the maximum perturbation amplitude; is the generated adversarial image; T is the input text of the model.

4. The method according to claim 1, wherein The step of generating common problems and improvement suggestions for the response content of the multimodal large language model based on the response content of the multimodal large language model for each malicious problem, and generating the guiding prompt words for the next round based on the common problems and the improvement suggestions includes: Using the first intelligent agent to score the response content corresponding to each malicious problem and output modification suggestions for each response content; Using the second intelligent agent to obtain the common problems and improvement suggestions for the response content of the multimodal large language model based on the scores of each response content, the modification suggestions corresponding to each response content, and the guiding prompt words of the current round; Using the third intelligent agent to generate the guiding prompt words for the next round based on the common problems, the improvement suggestions, and the guiding prompt words of the current round.

5. The method according to claim 4, wherein The step of scoring the response content corresponding to each malicious question by the first agent and outputting modification suggestions for each response content can be represented by the following formula: wherein, is the i-th harmful problem, is the response content of the model to the i-th harmful problem; is the score of the response content of the first agent to the i-th harmful problem, is the modification suggestion generated by the first agent for the i-th harmful problem and the corresponding response content.

6. The method according to claim 4, wherein The step of using the second agent to obtain the common problems and improvement suggestions for the response content of the multi-modal large language model based on the scores of each response content, the modification suggestions corresponding to each response content, and the guiding prompt words of the current round can be represented by the following formula: Among them, M represents common problems, U is an improvement suggestion; is the guiding prompt word for the previous round.

7. The method according to claim 4, characterized in that, The step of using the third agent to generate the guiding prompt words for the next round based on the common problems, the improvement suggestions, and the guiding prompt words of the current round can be represented by the following formula: Among them, is the k guidance prompt generated for the target at the 8. An apparatus for generating adversarial samples of a multimodal large language model that synergistically combines image perturbation and text guidance, characterized in that, The device includes: An image perturbation module, configured to add perturbations to the target image by using the malicious questions in the malicious question set and the guiding prompt words of the current round to generate an adversarial image, and input the adversarial image, the malicious questions in the malicious question set, and the guiding words of the current round into the multi-modal large language model to obtain the response content of the multi-modal large language model for each malicious question; the malicious question set contains multiple malicious questions; the target image is: a given image for generating the adversarial image; A text guiding module, configured to generate the common problems and improvement suggestions for the response content of the multi-modal large language model based on the response content of the multi-modal large language model for each malicious question, and generate the guiding prompt words for the next round based on the common problems and the improvement suggestions; A sample generation module, configured to use the guiding prompt words of the next round to iterate the adversarial image and the guiding prompt words until a preset end condition is met, and then generate an adversarial sample; Wherein, in each iteration, the perturbation of the target image is updated, a new adversarial image is generated, and the response content generated by the multi-modal large language model is guided by using the newly generated guiding prompt words based on the newly generated adversarial image; the adversarial sample includes: guiding prompt words and an adversarial image.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the steps of the method for generating an adversarial sample of a multi-modal large language model with coordinated image perturbation and text guidance as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer program is stored thereon. When the computer program is executed by a processor, it implements the steps of the method for generating an adversarial sample of a multi-modal large language model with coordinated image perturbation and text guidance as described in any one of claims 1 to 7.

Images