Multimodal large model black box defense method and device based on reinforcement learning

Abstract

The invention provides a multi-modal large model black box defense method and device based on reinforcement learning, and the method is characterized in that the method comprises the following steps: S1, purifying a jail break image through an image purifier to generate a purified image, and purifying a jail break text through a text purifier to generate a purified text; s2, inputting the purified text into a benign guide generator to generate a guide text; s3, splicing the purified text and the guide text to obtain a spliced text; and S4, inputting the spliced text and the purified image into a multi-modal large model together to generate a benign reply. In a word, the method can improve the defense capability of the multi-mode large model for the jailbreak attack, and generates a corresponding benign reply for the input.

Description

Technical Field

[0001] The present invention belongs to the field of model protection in the field of artificial intelligence security, and specifically relates to a multimodal large model black box defense method and device based on reinforcement learning. Background Art

[0002] With the rapid development of large language models integrating multimodality, multimodal large models have emerged. Representative applications include OpenAI's GPT-4o and Google's Gemini 1.5. Multimodal large models combine visual and textual modalities, and can achieve multiple tasks such as image annotation and visual question answering, greatly expanding the application scope of traditional large language models. However, the introduction of this multimodality has created more model security risks, especially multimodal large models are vulnerable to cross-modal jailbreak attacks. Jailbreak attacks use multimodal inputs with potential risks to disrupt the normal operation of multimodal large models. Therefore, in order to ensure the application security of multimodal large models in real-world scenarios, it is crucial to solve the above security risks.

[0003] Existing defense methods against jailbreak attacks on large multimodal models can be roughly divided into two categories: (1) white-box defense, which makes large multimodal models more robust in parameter space through adversarial training or fine-tuning; (2) black-box defense, which protects models through filters, detectors, or security-driven system prompts. Relatively speaking, black-box defense is more flexible and practical than white-box defense because it can protect the target large multimodal model without accessing model parameters.

[0004] Existing black-box defense methods are mainly divided into unimodal and bimodal defense. Unimodal defense focuses on defending text or visual cues. However, unimodal defense can only protect a single modality of a multimodal large model, and therefore cannot fully utilize the multimodal information in the input. Bimodal defense can address the limitations of unimodality, but will have a negative impact on the response to benign queries.

[0005] In short, there is still a lot of room for improvement in existing black box defense methods. Summary of the invention

[0006] The present invention is made to solve the above-mentioned problems, and its purpose is to provide a multimodal large model black box defense method and device based on reinforcement learning.

[0007] The present invention provides a multimodal large model black box defense method based on reinforcement learning, which is used to enable the multimodal large model to generate a benign response with such characteristics according to a jailbreak text and a corresponding jailbreak image, comprising the following steps: step S1, purifying the jailbreak image by an image purifier to generate a purified image, and purifying the jailbreak text by a text purifier to generate a purified text; step S2, inputting the purified text into a benign guidance generator to generate a guidance text; step S3, splicing the purified text and the guidance text to obtain a spliced ​​text; step S4, inputting the spliced ​​text and the purified image together into the multimodal large model to generate a benign response.

[0008] In the multimodal large model black box defense method based on reinforcement learning provided by the present invention, it can also have the following characteristics: wherein, the image purifier gradually adds noise to the jailbreak image using a plurality of time steps through a diffusion equation to obtain a highly noisy image, and iteratively denoises the highly noisy image through reverse diffusion to obtain a purified image, and the expression of the diffusion equation is: Where x s is the jailbreak image with added noise corresponding to time step s, α s is the control weight for controlling the amount of noise added at time step s, is the jailbreak image before adding noise corresponding to time step s, ∈ is the noise sampled from the standard normal distribution, the highly noisy image is the jailbreak image after adding noise corresponding to the last time step S, and the expression of reverse diffusion is: s-1 =f θ (x s ,s),s=S,S-1,…,1, where x s-1 is the denoised image generated at time step s, f θ is a denoising function controlled by parameter θ, and the highly noisy image is x S , the purified image is the denoised image corresponding to time step s=1.

[0009] In the multimodal large model black box defense method based on reinforcement learning provided by the present invention, it can also have the following characteristics: wherein the text purifier is an existing large language model, a preset rewriting template is combined with the jailbreak text and input into the large language model, and a rewriting text without changing the meaning of the jailbreak text is obtained as the purified text, and the purified text satisfies the following expression: In the formula To purify the text, y * Non-benign responses generated by a multimodal large model based on purified text, Generating non-benign responses y for multimodal large models * probability.

[0010] In the multimodal large model black box defense method based on reinforcement learning provided by the present invention, it can also have the following characteristics: wherein, the construction process of the benign guide generator includes the following steps: step T1, using the existing lightweight large language model as the initial benign guide generator; step T2, constructing a training data set containing multiple training purified texts and corresponding training purified images according to the existing jailbreak query data; step T3, selecting training purified text from the training data set and inputting it into the initial benign guide generator to generate the corresponding training guide text, and splicing it with the training purified text to obtain the training spliced ​​text; step T4, inputting the training spliced ​​text and the corresponding training purified image into the multimodal large model to obtain the training response; step T5, judging the training response through the existing large language model to obtain a security score; step T6, fine-tuning the initial benign guide generator according to the security score; step T7, repeating steps T3 to T6 until the preset termination condition is reached, and then using the fine-tuned initial benign guide generator as the benign guide generator.

[0011] In the multimodal large model black box defense method based on reinforcement learning provided by the present invention, it can also have the following characteristics: wherein, the large language model is used to determine whether the training response is a benign response, if so, the security score is 1, if not, the security score is 0.

[0012] The present invention also provides a multimodal large model black box defense device based on reinforcement learning, which is used to enable the multimodal large model to generate a benign response according to the jailbreak text and the corresponding jailbreak image, and has the following characteristics, including: an image purification module, including an image purifier, which is used to purify the jailbreak image to generate a purified image; a text purification module, including a text purifier, which is used to purify the jailbreak text to generate a purified text; a guide text generation module, including a benign guide generator, which is used to generate a guide text according to the purified text; a splicing module, which is used to splice the purified text and the guide text to obtain a spliced ​​text; a benign response module, including a multimodal large model, which is used to input the spliced ​​text and the purified image into the multimodal large model together to generate a benign response.

[0013] Functions and Effects of the Invention

[0014] According to the multimodal large model black box defense method and device based on reinforcement learning involved in the present invention, because the jailbreak image and jailbreak text are purified respectively, the corresponding purified image and purified text with reduced toxicity are generated, and then the guide text is generated according to the purified text, which is spliced ​​with the purified text, and then input into the multimodal large model together with the purified image, so that the multimodal large model can further improve the defense capability against jailbreak attacks according to the guide text. Therefore, the multimodal large model black box defense method and device based on reinforcement learning of the present invention can improve the defense capability of the multimodal large model against jailbreak attacks and generate corresponding benign responses to the input. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a block diagram of a multimodal large model black box defense device based on reinforcement learning in an embodiment of the present invention;

[0016] Figure 2 is a schematic diagram of a process of constructing a benign guidance generator in an embodiment of the present invention;

[0017] Figure 3 It is a flowchart of a multimodal large model black box defense method based on reinforcement learning in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the following embodiments and the accompanying drawings specifically illustrate the multimodal large model black box defense method and device based on reinforcement learning of the present invention.

[0019] This embodiment provides a multimodal large model black box defense device based on reinforcement learning, which is used to enable a multimodal large model attacked by jailbreak to generate a benign response based on the jailbreak text and the corresponding jailbreak image. In this embodiment, a benign response is a response that does not contain harmful content, and the benign response corresponds to the question input to the large model, and is not a response that is irrelevant to the question.

[0020] Figure 1 It is a block diagram of a multimodal large model black box defense device based on reinforcement learning in an embodiment of the present invention.

[0021] like Figure 1 As shown, the multimodal large model black box defense device 100 based on reinforcement learning includes an image purification module 10, a text purification module 20, a guided text generation module 30, a splicing module 40 and a benign response module 50.

[0022] The image purification module 10 includes an image purifier, which is used to purify the jailbreak image to generate a purified image.

[0023] Among them, the image purifier gradually adds noise to the jailbreak image using multiple time steps through the diffusion equation to obtain a highly noisy image, and iteratively denoises the highly noisy image through reverse diffusion to obtain a purified image. That is, the image purifier gradually destroys the jailbreak image by gradually adding noise, and then gradually removes the noise to restore the clean sample, and finally obtains a purified image with reduced jailbreak capability.

[0024] The expression of the diffusion equation is:

[0025]

[0026] Where x sis the jailbreak image with added noise corresponding to time step s, α s is the control weight for controlling the amount of noise added at time step s, is the jailbreak image before adding noise corresponding to time step s, ∈ is the noise sampled from the standard normal distribution. The highly noisy image is the jailbreak image after adding noise corresponding to the last time step S.

[0027] The expression for back diffusion is:

[0028] x s-1 =f θ (x s ,s),s=S,S-1,…,1,

[0029] Where x s-1 is the denoised image generated at time step s, f θ is a denoising function controlled by parameter θ, and the highly noisy image is x S The purified image is the denoised image corresponding to the time step s=1.

[0030] The text purification module 20 includes a text purifier, which is used to purify the jailbreak text to generate purified text.

[0031] The text purifier is an existing large language model. The preset rewriting template is combined with the jailbreak text and input into the large language model to obtain the rewritten text without changing the meaning of the jailbreak text as the purified text. In this embodiment, GPT-4o is used as a text purifier to generate a purified text by rewriting the jailbreak text to add a more detailed description.

[0032] In this embodiment, the purified text is as similar to the jailbreak text as possible, and the jailbreak capability of the jailbreak text needs to be reduced, that is, the purified text satisfies the following expression:

[0033]

[0034] In the formula To purify the text, y * Non-benign responses generated by a multimodal large model based on purified text, Generating non-benign responses y for multimodal large models * probability.

[0035] The guide text generation module 30 includes a benign guide generator, which is used to generate a guide text according to the purified text.

[0036] Figure 2 It is a schematic diagram of the process of constructing a benign guidance generator in an embodiment of the present invention.

[0037] like Figure 2As shown in Figure 1, the construction process of the benign boot generator includes the following steps:

[0038] Step T1: Use the existing lightweight large language model as an initial benign guide generator. In this embodiment, the GPT-2 model is used as the initial benign guide generator.

[0039] Step T2: construct a training data set including a plurality of training cleansed texts and corresponding training cleansed images according to the existing jailbreak query data.

[0040] Step T3, selecting training purified text from the training data set and inputting it into the initial benign guide generator to generate corresponding training guide text, and concatenating it with the training purified text to obtain training concatenated text.

[0041] Step T4: input the training concatenated text and the corresponding training purified image into the multimodal large model to obtain the training response.

[0042] Step T5: Use the existing large language model to judge the training response and obtain a safety score. The large language model is used to judge whether the training response is a benign response. If so, the safety score is 1, and if not, the safety score is 0.

[0043] Step T6, fine-tune the initial benign guidance generator according to the security score.

[0044] Step T7, repeat steps T3 to T6 until a preset termination condition is reached, and then the fine-tuned initial benign guide generator is used as the benign guide generator.

[0045] In this embodiment, since the security score is obtained based on the responses generated by the multimodal large model through image input and text input, the initial benign guidance generator is fine-tuned according to the security score. The final fine-tuned initial benign guidance generator, i.e., the guidance text generated by the benign guidance generator, can take into account the information of both image and text modes, thereby helping to enhance the cross-modal robustness of the model.

[0046] The splicing module 40 is used to splice the cleaned text and the guide text to obtain a spliced ​​text. In this embodiment, the guide text is used as a text suffix and added to the end of the cleaned text to achieve splicing.

[0047] The benign reply module 50 includes a multimodal large model, which is used to input the spliced ​​text and the cleaned image into the multimodal large model together to generate a benign reply.

[0048] The following, in conjunction with the accompanying drawings, describes the process of using the reinforcement learning-based multimodal large model black box defense device 100 to perform the reinforcement learning-based multimodal large model black box defense method.

[0049] Figure 3 It is a flowchart of a multimodal large model black box defense method based on reinforcement learning in an embodiment of the present invention.

[0050] like Figure 3 As shown in the figure, the multimodal large model black box defense method based on reinforcement learning includes the following steps:

[0051] Step S1, using the image purification module 10 to purify the jailbreak image to generate a purified image, and using the text purification module 20 to purify the jailbreak text to generate a purified text.

[0052] Step S2, using the guide text generation module 30 to input the purified text into the benign guide generator to generate a guide text.

[0053] Step S3, using the splicing module 40 to splice the purified text and the guide text to obtain a spliced ​​text.

[0054] Step S4, using the benign response module 50 to input the concatenated text and the purified image into the multimodal large model to generate a benign response.

[0055] In this embodiment, the multimodal large model black box defense method based on reinforcement learning is used to perform jailbreak attack defense tests on the open source large models LLaVA and MiniGPT-4 and the commercial large model Gemini. The interception success rate on the open source large models LLaVA and MiniGPT-4 is higher than 90%, and the jailbreak interception success rate on the commercial large model Gemini is higher than 99%. It can be seen that the multimodal large model black box defense method based on reinforcement learning in this embodiment can effectively defend against jailbreak attacks against multimodal large models.

[0056] Functions and Effects of the Embodiments

[0057] According to the multimodal large model black box defense method and device based on reinforcement learning involved in this embodiment, the jailbreak image and jailbreak text are purified respectively to generate corresponding purified images and purified texts with reduced toxicity, and then a guide text is generated based on the purified text, which is spliced ​​with the purified text and then input into the multimodal large model together with the purified image, so that the multimodal large model can further improve its defense capability against jailbreak attacks based on the guide text. In short, this method can improve the defense capability of the multimodal large model against jailbreak attacks and generate corresponding benign responses to the input.

[0058] Those skilled in the art should understand that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A multimodal large model black box defense method based on reinforcement learning, used to enable the multimodal large model to generate benign responses based on jailbreak text and corresponding jailbreak images, characterized in that: The following steps are involved: Step S1, purifying the jailbreak image by an image purifier to generate a purified image, and purifying the jailbreak text by a text purifier to generate a purified text; Step S2, inputting the purified text into a benign guidance generator to generate a guidance text; Step S3, splicing the purified text and the guide text to obtain a spliced ​​text; Step S4, inputting the concatenated text and the purified image into the multimodal large model to generate the benign response.

2. The multimodal large model black box defense method based on reinforcement learning according to claim 1 is characterized in that: in, The image purifier gradually adds noise to the jailbreak image using a diffusion equation using multiple time steps to obtain a highly noisy image, and iteratively denoises the highly noisy image through reverse diffusion to obtain the purified image. The expression of the diffusion equation is: Where x s is the jailbreak image with added noise corresponding to time step s, α s is the control weight for controlling the amount of noise added at time step s, is the jailbreak image before adding noise corresponding to time step s, ∈ is the noise sampled from the standard normal distribution, The highly noisy image is the jailbreak image with noise added corresponding to the last time step S. The expression of the reverse diffusion is: x s-1 =f θ (x s ,s),s=S,S-1,…,1, Where x s-1 is the denoised image generated at time step s, f θ is a denoising function controlled by parameter θ, and the highly noisy image is x S , The purified image is the denoised image corresponding to the time step s=1.

3. The multimodal large model black box defense method based on reinforcement learning according to claim 1 is characterized in that: in, The text purifier is an existing large language model. Combining a preset rewriting template with the jailbreak text and inputting the template into the large language model, obtaining a rewriting text that does not change the meaning of the jailbreak text as the purified text, The purified text satisfies the following expression: In the formula For the purified text, y * The multimodal model is based on the purified text The non-benign responses generated, Generate non-benign responses y for the multimodal large model * probability.

4. The multimodal large model black box defense method based on reinforcement learning according to claim 1, Features: in, The construction process of the benign boot generator includes the following steps: Step T1, using the existing lightweight large language model as the initial benign guidance generator; Step T2, constructing a training data set including a plurality of training cleansed texts and corresponding training cleansed images according to the existing jailbreak query data; Step T3, selecting a training cleansed text from the training data set and inputting it into the initial benign guide generator to generate a corresponding training guide text, and concatenating it with the training cleansed text to obtain a training concatenated text; Step T4, inputting the training concatenated text and the corresponding training purified image into the multimodal large model to obtain a training response; Step T5, judging the training response through the existing large language model to obtain a safety score; Step T6, fine-tuning the initial benign guidance generator according to the security score; Step T7, repeating steps T3 to T6 until a preset termination condition is reached, and then using the fine-tuned initial benign guide generator as the benign guide generator.

5. The multimodal large model black box defense method based on reinforcement learning according to claim 4 is characterized in that: in, The large language model is used to determine whether the training response is a benign response. If so, the safety score is 1; if not, the safety score is 0.

6. A multimodal large model black box defense device based on reinforcement learning, used to enable the multimodal large model to generate a benign response based on the jailbreak text and the corresponding jailbreak image, characterized in that: include: An image purification module, including an image purifier, for purifying the jailbreak image to generate a purified image; A text purification module, including a text purifier, for purifying the jailbreak text to generate purified text; A guide text generation module, including a benign guide generator, for generating a guide text according to the purified text; A splicing module, used for splicing the purified text and the guide text to obtain a spliced ​​text; The benign reply module includes the multimodal large model and is used to input the spliced ​​text and the cleansed image into the multimodal large model to generate the benign reply.

Images