Multimodal large model black box defense method and device based on reinforcement learning
By purifying jailbreak images and text and guiding text generation, the problem that existing black-box defense methods cannot fully utilize multimodal information is solved, and effective defense and benign response generation for large multimodal models are achieved.
Patent Information
- Application Number
- CN202411503194.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-25
- Publication Date
- 2025-09-30
- Estimated Expiration
- 2044-10-25
AI Technical Summary
Existing black-box defense methods cannot fully utilize multimodal information, have a negative impact on responses to benign queries, and cannot effectively defend against jailbreak attacks on large multimodal models.
The jailbreak images and texts are purified by the image purifier and text purifier to generate purified images and purified texts, and the benign guidance generator is used to generate guidance texts, which are then spliced and input into a large multimodal model to generate benign responses.
The multimodal large model's defense capability against jailbreak attacks is improved, corresponding benign responses are generated, and the risk of jailbreak attacks is reduced.
Smart Images
Figure CN119449397B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of model protection in the field of artificial intelligence security, and specifically relates to a multimodal large-model black box defense method and device based on reinforcement learning. Background Art
[0002] With the rapid development of multimodal integration in large language models, large multimodal models have emerged. Representative applications include OpenAI's GPT-4o and Google's Gemini 1.5. These large multimodal models combine visual and textual modalities, enabling a variety of tasks such as image annotation and visual question answering, significantly expanding the application scope of traditional large language models. However, the introduction of multimodality creates additional model security risks. In particular, large multimodal models are vulnerable to cross-modal jailbreak attacks. Jailbreak attacks exploit potentially risky multimodal inputs to disrupt the normal operation of large multimodal models. Therefore, addressing these security risks is crucial to ensuring the secure application of large multimodal models in real-world scenarios.
[0003] Existing defense methods against jailbreaking attacks on large multimodal models can be roughly divided into two categories: (1) white-box defense, which makes large multimodal models more robust in parameter space through adversarial training or fine-tuning; and (2) black-box defense, which protects models through filters, detectors, or security-driven system prompts. Relatively speaking, black-box defense is more flexible and practical than white-box defense because it can protect the target large multimodal model without accessing the model parameters.
[0004] Existing black-box defense methods are primarily categorized as unimodal and bimodal. Unimodal defenses focus on defending against textual or visual cues. However, unimodal defenses only protect a single modality of a large multimodal model, thus failing to fully exploit the multimodal information in the input. Bimodal defenses address the limitations of unimodality but can negatively impact responses to benign queries.
[0005] In short, there is still much room for improvement in existing black box defense methods. Summary of the Invention
[0006] The present invention is made to solve the above-mentioned problems, and its purpose is to provide a multimodal large model black box defense method and device based on reinforcement learning.
[0007] The present invention provides a multimodal large model black box defense method based on reinforcement learning, which is used to enable the multimodal large model to generate a benign response with such characteristics based on jailbreak text and corresponding jailbreak images, comprising the following steps: step S1, purifying the jailbreak image by an image purifier to generate a purified image, and purifying the jailbreak text by a text purifier to generate a purified text; step S2, inputting the purified text into a benign guidance generator to generate a guidance text; step S3, splicing the purified text and the guidance text to obtain a spliced text; step S4, inputting the spliced text and the purified image together into the multimodal large model to generate a benign response.
[0008] The multimodal large-model black-box defense method based on reinforcement learning provided by the present invention may also have the following features: wherein, the image purifier gradually adds noise to the jailbreak image using a diffusion equation in multiple time steps to obtain a highly noisy image, and iteratively denoises the highly noisy image by back diffusion to obtain a purified image. The expression of the diffusion equation is: Where x s is the jailbreak image with added noise corresponding to time step s, α s is the control weight for controlling the amount of noise added at time step s, is the jailbreak image before adding noise corresponding to time step s, ∈ is the noise sampled from the standard normal distribution, the highly noisy image is the jailbreak image after adding noise corresponding to the last time step S, and the expression of back diffusion is: s-1 =f θ (x s ,s),s=S,S-1,…,1, where x s-1 The denoised image generated for time step s, f θ is a denoising function controlled by parameter θ, and the highly noisy image is x S , the purified image is the denoised image corresponding to time step s=1.
[0009] The multimodal large-model black-box defense method based on reinforcement learning provided by the present invention may also have the following features: wherein the text purifier is an existing large language model, a preset rewriting template is combined with the jailbreak text and input into the large language model, and the rewritten text that does not change the meaning of the jailbreak text is obtained as the purified text, and the purified text satisfies the following expression: In the formula To purify the text, y * Non-benign responses generated by a large multimodal model based on purified text, Generate non-benign responses y for multimodal large models * probability.
[0010] The multimodal large model black box defense method based on reinforcement learning provided by the present invention may also have the following features: wherein, the construction process of the benign guide generator includes the following steps: step T1, using the existing lightweight large language model as the initial benign guide generator; step T2, constructing a training data set containing multiple training purified texts and corresponding training purified images based on the existing jailbreak query data; step T3, selecting training purified text from the training data set and inputting it into the initial benign guide generator to generate corresponding training guide text, and splicing it with the training purified text to obtain training spliced text; step T4, inputting the training spliced text and the corresponding training purified image into the multimodal large model to obtain training responses; step T5, judging the training responses through the existing large language model to obtain a security score; step T6, fine-tuning the initial benign guide generator according to the security score; step T7, repeating steps T3 to T6 until the preset termination condition is reached, and then using the fine-tuned initial benign guide generator as the benign guide generator.
[0011] The multimodal large-model black-box defense method based on reinforcement learning provided by the present invention may also have the following features: wherein, the large language model is used to determine whether the training response is a benign response. If so, the security score is 1, and if not, the security score is 0.
[0012] The present invention also provides a multimodal large model black box defense device based on reinforcement learning, which is used to enable the multimodal large model to generate benign responses based on jailbreak text and corresponding jailbreak images. The device has the following characteristics: an image purification module, including an image purifier, which is used to purify the jailbreak image to generate a purified image; a text purification module, including a text purifier, which is used to purify the jailbreak text to generate purified text; a guide text generation module, including a benign guide generator, which is used to generate a guide text based on the purified text; a splicing module, which is used to splice the purified text and the guide text to obtain a spliced text; a benign response module, including a multimodal large model, which is used to input the spliced text and the purified image into the multimodal large model to generate a benign response.
[0013] Functions and effects of the invention
[0014] According to the reinforcement learning-based multimodal large-model black-box defense method and device involved in the present invention, because the jailbreak image and jailbreak text are purified separately to generate corresponding purified images and purified text with reduced toxicity, a guide text is generated based on the purified text, which is spliced with the purified text and then input into the multimodal large model together with the purified image, thereby enabling the multimodal large model to further improve its defense capabilities against jailbreak attacks based on the guide text. Therefore, the reinforcement learning-based multimodal large-model black-box defense method and device of the present invention can improve the multimodal large model's defense capabilities against jailbreak attacks and generate corresponding benign responses to the input. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 is a block diagram of a multimodal large model black box defense device based on reinforcement learning in an embodiment of the present invention;
[0016] Figure 2 is a schematic diagram of a process for constructing a benign guidance generator in an embodiment of the present invention;
[0017] Figure 3 2 is a flowchart of a multimodal large model black box defense method based on reinforcement learning in an embodiment of the present invention. DETAILED DESCRIPTION
[0018] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the following embodiments and the accompanying drawings specifically illustrate the multimodal large model black box defense method and device based on reinforcement learning of the present invention.
[0019] This embodiment provides a reinforcement learning-based multimodal large-scale black-box defense device, designed to enable a multimodal large-scale model vulnerable to jailbreak attacks to generate benign responses based on jailbreak text and corresponding jailbreak images. In this embodiment, a benign response is one that does not contain harmful content and corresponds to the question posed to the large-scale model, rather than being irrelevant to the question.
[0020] Figure 1 4 is a block diagram of a multimodal large-model black-box defense device based on reinforcement learning in an embodiment of the present invention.
[0021] like Figure 1 As shown, the multimodal large model black box defense device 100 based on reinforcement learning includes an image purification module 10, a text purification module 20, a guided text generation module 30, a splicing module 40 and a benign response module 50.
[0022] The image purification module 10 includes an image purifier, which is used to purify the jailbreak image to generate a purified image.
[0023] The image purifier uses a diffusion equation to gradually add noise to the jailbreak image over multiple time steps to produce a highly noisy image. It then iteratively denoises the highly noisy image using inverse diffusion to produce a purified image. In other words, the image purifier gradually degrades the jailbreak image by adding noise, then gradually removes the noise to restore a clean sample, ultimately producing a purified image that reduces jailbreak capabilities.
[0024] The expression of the diffusion equation is:
[0025]
[0026] Where x sis the jailbreak image with added noise corresponding to time step s, α s is the control weight for controlling the amount of noise added at time step s, is the jailbreak image before adding noise corresponding to time step s, ∈ is the noise sampled from the standard normal distribution. The highly noisy image is the jailbreak image after adding noise corresponding to the last time step S.
[0027] The expression for back diffusion is:
[0028] x s-1 =f θ (x s ,s),s=S,S-1,…,1,
[0029] Where x s-1 The denoised image generated for time step s, f θ is a denoising function controlled by parameter θ, and the highly noisy image is x S The purified image is the denoised image corresponding to time step s=1.
[0030] The text purification module 20 includes a text purifier for purifying the jailbreak text to generate purified text.
[0031] The text purifier is an existing large language model. A pre-set rewriting template is combined with the jailbreak text and then fed into the large language model to produce a rewritten text that does not change the meaning of the jailbreak text, which serves as the purified text. In this embodiment, GPT-4o is used as the text purifier, rewriting the jailbreak text to add a more detailed description to generate the purified text.
[0032] In this embodiment, the purified text is as similar as possible to the jailbreak text, while also reducing the jailbreak capability of the jailbreak text. That is, the purified text satisfies the following expression:
[0033]
[0034] In the formula To purify the text, y * Non-benign responses generated by a large multimodal model based on purified text, Generate non-benign responses y for multimodal large models * probability.
[0035] The guidance text generation module 30 includes a benign guidance generator, which is used to generate a guidance text according to the purified text.
[0036] Figure 2 It is a schematic diagram of the process of constructing a benign guidance generator in an embodiment of the present invention.
[0037] like Figure 2As shown in Figure 2, the construction process of the benign boot generator includes the following steps:
[0038] In step T1, the existing lightweight large language model is used as the initial benign guidance generator. In this embodiment, the GPT-2 model is used as the initial benign guidance generator.
[0039] Step T2: construct a training dataset containing multiple training cleansed texts and corresponding training cleansed images based on the existing jailbreak query data.
[0040] Step T3: Select a training cleansed text from the training data set and input it into the initial benign guide generator to generate a corresponding training guide text, and then splice it with the training cleansed text to obtain a training spliced text.
[0041] In step T4, the training concatenated text and the corresponding training purified image are input into the multimodal large model to obtain the training response.
[0042] In step T5, the training response is judged by the existing large language model to obtain a safety score. The large language model is used to determine whether the training response is a benign response. If so, the safety score is 1; otherwise, the safety score is 0.
[0043] Step T6: fine-tune the initial benign guidance generator based on the security score.
[0044] Step T7, repeat steps T3 to T6 until the preset termination condition is reached, and then the fine-tuned initial benign guide generator is used as the benign guide generator.
[0045] In this embodiment, since the security score is obtained based on the responses generated by the multimodal large model through both image input and text input, the initial benign guidance generator is fine-tuned according to the security score. This allows the final fine-tuned initial benign guidance generator, i.e., the guidance text generated by the benign guidance generator, to take into account both image and text bimodal information, thereby helping to enhance the cross-modal robustness of the model.
[0046] The splicing module 40 is used to splice the cleaned text and the guide text to obtain a spliced text. In this embodiment, the guide text is added as a text suffix at the end of the cleaned text to achieve splicing.
[0047] The good reply module 50 includes a multimodal large model, which is used to input the spliced text and the cleansed image into the multimodal large model to generate a good reply.
[0048] The following describes the process of using the reinforcement learning-based multimodal large model black box defense device 100 to perform the reinforcement learning-based multimodal large model black box defense method in conjunction with the accompanying drawings.
[0049] Figure 3 2 is a flowchart of a multimodal large model black box defense method based on reinforcement learning in an embodiment of the present invention.
[0050] like Figure 3 As shown in the figure, the multimodal large model black box defense method based on reinforcement learning includes the following steps:
[0051] Step S1: Use the image purification module 10 to purify the jailbreak image to generate a purified image, and use the text purification module 20 to purify the jailbreak text to generate a purified text.
[0052] Step S2: Using the guidance text generation module 30, the cleansed text is input into the benign guidance generator to generate guidance text.
[0053] Step S3: Using the splicing module 40 to splice the cleaned text and the guide text to obtain a spliced text.
[0054] In step S4, the benign response module 50 is used to input the concatenated text and the purified image into the multimodal large model to generate a benign response.
[0055] In this embodiment, a multimodal large-scale model black-box defense method based on reinforcement learning was used to conduct jailbreak attack defense tests on the open-source large-scale models LLaVA and MiniGPT-4, as well as the commercial large-scale model Gemini. The interception success rate on the open-source large-scale models LLaVA and MiniGPT-4 was higher than 90%, and the jailbreak interception success rate on the commercial large-scale model Gemini was higher than 99%. This shows that the multimodal large-scale model black-box defense method based on reinforcement learning in this embodiment can effectively defend against jailbreak attacks targeting multimodal large-scale models.
[0056] Functions and Effects of the Embodiments
[0057] According to the reinforcement learning-based multimodal large-scale black-box defense method and device involved in this embodiment, jailbreak images and jailbreak text are purified separately to generate corresponding purified images and purified text with reduced toxicity. A guidance text is then generated based on the purified text, concatenated with the purified text, and then input into the multimodal large-scale model along with the purified image. This allows the multimodal large-scale model to further improve its defense against jailbreak attacks based on the guidance text. In short, this method can improve the multimodal large-scale model's defense capabilities against jailbreak attacks and generate corresponding benign responses to inputs.
[0058] Those skilled in the art will appreciate that the present invention is not limited to the foregoing embodiments. The foregoing embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A multimodal large model black box defense method based on reinforcement learning, which is used to enable the multimodal large model to generate benign responses based on jailbreak text and corresponding jailbreak images, characterized in that: The following steps are involved: Step S1, purifying the jailbreak image by an image purifier to generate a purified image, and purifying the jailbreak text by a text purifier to generate a purified text; Step S2, inputting the cleansed text into a benign guidance generator to generate a guidance text; Step S3, splicing the purified text and the guide text to obtain a spliced text; Step S4: input the concatenated text and the cleansed image into the multimodal large model to generate the benign response. The construction process of the benign guidance generator includes the following steps: Step T1: Use the existing lightweight large language model as the initial benign guidance generator; Step T2: constructing a training dataset containing multiple training cleansed texts and corresponding training cleansed images based on the existing jailbreak query data; Step T3, selecting a training cleansed text from the training data set and inputting it into the initial benign guide generator to generate a corresponding training guide text, and then splicing it with the training cleansed text to obtain a training spliced text; Step T4: input the training concatenated text and the corresponding training cleansed image into the multimodal large model to obtain a training response; Step T5: Using the existing large language model to judge the training response and obtain a safety score; Step T6, fine-tuning the initial benign guidance generator according to the security score; Step T7, repeating steps T3 to T6 until a preset termination condition is reached, and then using the fine-tuned initial benign guide generator as the benign guide generator.
2. The multimodal large model black box defense method based on reinforcement learning according to claim 1 is characterized by: in, The image purifier gradually adds noise to the jailbreak image using a diffusion equation in multiple time steps to obtain a highly noisy image, and iteratively denoises the highly noisy image through inverse diffusion to obtain the purified image. The diffusion equation is expressed as: Where x s is the jailbreak image with added noise corresponding to time step s, α s is the control weight for controlling the amount of noise added at time step s, is the jailbreak image before adding noise corresponding to time step s, ∈ is the noise sampled from the standard normal distribution, The highly noisy image is the jailbreak image with noise added corresponding to the last time step S. The expression of the reverse diffusion is: x s-1 =f θ (x s ,s),s=S,S-1,…,1, Where x s-1 The denoised image generated for time step s, f θ is a denoising function controlled by parameter θ, and the highly noisy image is x S , The purified image is the denoised image corresponding to the time step s=1.
3. The multimodal large model black box defense method based on reinforcement learning according to claim 1 is characterized by: in, The text purifier is an existing large language model, Combining a preset rewriting template with the jailbreak text and inputting the template into the large language model, obtaining a rewritten text that does not change the meaning of the jailbreak text as the purified text, The purified text satisfies the following expression: In the formula For the purified text, y * The multimodal model is based on the purified text The generated non-benign responses, Generate non-benign responses y for the multimodal large model * probability.
4. The multimodal large model black box defense method based on reinforcement learning according to claim 1 is characterized by: in, The large language model is used to determine whether the training response is a benign response. If so, the safety score is 1; if not, the safety score is 0.
5. A multimodal large model black box defense device based on reinforcement learning, used to enable the multimodal large model to generate benign responses based on jailbreak text and corresponding jailbreak images, characterized in that: include: An image purification module, comprising an image purifier, configured to purify the jailbreak image to generate a purified image; A text purification module, including a text purifier, for purifying the jailbreak text to generate purified text; A guide text generation module, comprising a benign guide generator, configured to generate a guide text based on the purified text; A splicing module, configured to splice the purified text and the guide text to obtain a spliced text; A benign reply module, comprising the multimodal large model, is configured to input the concatenated text and the cleansed image into the multimodal large model to generate the benign reply. The construction process of the benign guidance generator includes the following steps: Step T1: Use the existing lightweight large language model as the initial benign guidance generator; Step T2: constructing a training dataset containing multiple training cleansed texts and corresponding training cleansed images based on the existing jailbreak query data; Step T3, selecting a training cleansed text from the training data set and inputting it into the initial benign guide generator to generate a corresponding training guide text, and then splicing it with the training cleansed text to obtain a training spliced text; Step T4: input the training concatenated text and the corresponding training cleansed image into the multimodal large model to obtain a training response; Step T5: Using the existing large language model to judge the training response and obtain a safety score; Step T6, fine-tuning the initial benign guidance generator according to the security score; Step T7, repeating steps T3 to T6 until a preset termination condition is reached, and then using the fine-tuned initial benign guide generator as the benign guide generator.