Video frame-based video multi-mode large model jailbreak attack method, system and equipment and medium

Abstract

The invention discloses a video multi-mode large model jailbreak attack method, system and device based on video frames and a medium. The method comprises the following steps: constructing a harmful corpus; decomposing the video into video frames, generating noise by using a harmful corpus, adding the noise to the video frames, generating antagonistic video frames, and synthesizing all the generated antagonistic video frames into a video jailbreak sample; finding an optimal harmful text instruction through a fixed video jailbreak sample; and combining the video jailbreak sample and the optimal harmful text instruction as joint input of the video multi-modal large model, so as to evaluate the safety performance of the video multi-modal large model to be evaluated. According to the method, a universal video multi-mode large model after video jailbreak attack sample attack security alignment can be effectively generated, and resource consumption caused by jailbreak attack of a plurality of different multi-mode large models is remarkably reduced.

Description

Technical Field

[0001] The present invention relates to the field of adversarial learning, and in particular to a video frame-based video multimodal large model jailbreak attack method, system, device and medium. Background Art

[0002] Multimodal big models refer to large-scale machine learning models that can process and understand multiple types of data, such as text, images, audio, video, and other data. These models usually use a unified architecture to fuse information from different modalities, so that they can leverage cross-modal relationships to improve performance in complex tasks. Multimodal big models aim to achieve more comprehensive and in-depth understanding and analysis by integrating different types of data. Unlike traditional single-modal big models, multimodal big models can process multiple data sources simultaneously, capture richer information, and improve the robustness and accuracy of the model.

[0003] Jailbreak attacks usually refer to attacks on models to make them deviate from their expected behavior, especially to make the model perform functions that it is not designed to allow or generate inappropriate content. Such attacks mainly target natural language processing and multimodal models.

[0004] Despite the rapid development of multimodal large model technology, models still have the problem of behavior that deviates from the original intention of the designer, generating incorrect or even potentially harmful outputs. Video multimodal large models are more susceptible to such potential harms due to their short development time. In order to address and reduce such potential security risks, jailbreak attacks have become a key red team strategy to bypass the model's protection mechanism and evaluate the model's robustness under abnormal input conditions. Once the model is successfully jailbroken, the attacker will manipulate the model to bypass the security line to generate harmful outputs, posing serious security risks. At present, jailbreak attack methods for multimodal large models mainly focus on customizing specific jailbreak perturbations for each image modality.

[0005] However, this jailbreak attack method has some major flaws. First, once the jailbreak perturbation is detected, the jailbreak attack effect will be significantly weakened; second, retraining the perturbation for each attack scenario will incur high computational costs. In addition, the scalability of existing methods is significantly restricted when attacking large multimodal video models. These defects emphasize the necessity of a general multimodal large model jailbreak attack sample to ensure that an efficient jailbreak attack sample that can attack multiple multimodal large models including video multimodal large models can be generated under the premise of low resource consumption, further improving the robustness and security of multimodal large models in various complex situations. Summary of the invention

[0006] In view of the deficiencies in the prior art, the present invention provides a video multimodal large model jailbreak attack method, system, device and medium based on video frames.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] A video multimodal large model jailbreak attack method based on video frames, comprising the following steps:

[0009] Step 1: Build a harmful corpus;

[0010] Step 2: Decompose the video into video frames, use the harmful corpus to generate noise and add it to the video frames to generate adversarial video frames, and synthesize all the generated adversarial video frames into video jailbreak samples;

[0011] Step 3: Find the optimal harmful text instructions by fixing the video jailbreak sample;

[0012] Step 4: Use the video jailbreak sample and the optimal harmful text instruction combination as the joint input of the video multimodal large model to evaluate the security performance of the video multimodal large model to be evaluated.

[0013] To optimize the above technical solutions, the specific measures taken also include:

[0014] Furthermore, step 1 is specifically as follows:

[0015] Process the M pieces of harmful text data containing discrimination collected and build a harmful corpus without query relevance Used for optimization of video jailbreak samples, represents the mth harmful target text, and M represents the total number of harmful target texts.

[0016] Furthermore, step 2 is specifically as follows:

[0017] Step 2.1: Decompose the video into a set of video frames and from the video frame set Sample a batch of video frames B j , where N frame is the total number of video frames, Represents the i-th video frame;

[0018] Step 2.2: batch video frames B j Each video frame sample in Select a corresponding harmful target text from the harmful corpus for each video frame sample is the kth video frame, x k is the text prompt corresponding to the k-th video frame;

[0019] The perturbed video frame samples Input to the visual encoder G v 、Connector Gc and the large language model G l The output result is obtained from the combination of:

[0020]

[0021] In the formula, y k It represents the video content recognized by the large language model for the k-th video frame with disturbance added;

[0022] Step 2.3: Use the large language model to identify the video content y k and harmful target text Compute the gradient with respect to the random noise ζ:

[0023]

[0024] in, represents the loss function, represents the kth harmful target text, represents the partial derivative of the loss function with respect to the noise ζ, g k represents the gradient update value for the noise ζ;

[0025] Step 2.4: Update noise Where a represents the learning rate, sign(·,·) represents the sign operation used to preserve the gradient direction, k is the index of the current sample, |B j | represents a batch of video frames B obtained by sampling j The total number of samples in ;

[0026] Step 2.5, clip the noise ζ = CLIP(ζ, -ε, ε), and limit the disturbance value between the lower limit of disturbance -ε and the upper limit of disturbance ε. CLIP(·) is a clipping mechanism for limiting the amplitude of noise ζ;

[0027] Step 2.6: Repeat steps 2.2 to 2.5 until the maximum number of iterations T is reached, and the noise with the best jailbreak attack effect is obtained, and added to the video frame to generate an adversarial video frame;

[0028] Step 2.7: Perform steps 2.1 to 2.6 for each frame of the video split, and finally synthesize all the generated adversarial video frames into a video jailbreak sample x adv .

[0029] Furthermore, step 3 is specifically as follows:

[0030] Step 3.1: Given a harmful intent q as the initial harmful text instruction, Jailbreak video sample x adv and initial harmful text instructions Input the video multimodal large model to obtain the initial video content recognition result; F θ (·) represents the process of video multimodal large model recognizing video content;

[0031] Step 3.2: Use a judgment prompt word template P J The second largest model M is used to evaluate the video content recognition result of the video multimodal large model; the second largest model M outputs a judgment result flag=M(P J ,q,y (i) ),in represents the output of the video multimodal large model, Indicates the harmful text instructions of the i-th iteration round. If flag returns 1, it means the jailbreak attack is successful; otherwise, flag returns 0;

[0032] Step 3.3: Use the prompt template P based on the thought chain C and harmful intent q, the second largest model M is based on the output y of the video multimodal large model (i) and the latest harmful text instructions Analyze the reasons for the latest jailbreak failure; then use the zero-sample form of the thought chain to C Provide a restatement strategy to assist in optimization. Re-expressed into new harmful text instructions The specific formula is:

[0033]

[0034] In the formula, M represents the second largest model;

[0035] Determine whether the maximum number of iterations N has been reached. If not, return to step 3.2. If so, obtain the optimal harmful text instruction x. harm .

[0036] Furthermore, step 4 is specifically as follows:

[0037] The final video jailbreak sample x adv and optimal harmful text instruction x harm The combination is used as the joint input p(g|[x adv ,x harm ]) to evaluate the security performance of the large multimodal video model to be evaluated.

[0038] The present invention also proposes a video multimodal large model jailbreak attack system based on video frames, comprising:

[0039] Harmful corpus construction module, used to construct harmful corpus;

[0040] A video jailbreak sample generation module is used to decompose the video into video frames, generate noise using a harmful corpus and add it to the video frames to generate adversarial video frames, and synthesize all generated adversarial video frames into video jailbreak samples;

[0041] The optimal harmful text instruction generation module is used to find the optimal harmful text instruction by fixing the video jailbreak sample;

[0042] The evaluation module is used to take the video jailbreak sample and the optimal harmful text instruction combination as the joint input of the video multimodal large model to evaluate the security performance of the video multimodal large model to be evaluated.

[0043] The present invention also proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the video multimodal large model jailbreak attack method based on video frames as described above is implemented.

[0044] The present invention also proposes a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the video frame-based video multimodal large model jailbreak attack method as described above.

[0045] The beneficial effects of the present invention are as follows: the present invention can effectively generate a universal video jailbreak attack sample attack security aligned video multimodal large model, significantly reducing the resource consumption caused by jailbreak attack on multiple different multimodal large models, further improving the success rate of jailbreak attack video multimodal large models, and emphasizing the necessity of developing more effective alignment and defense mechanisms in a multimodal context. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flow chart of the video frame-based video multimodal large model jailbreak attack method proposed by the present invention. DETAILED DESCRIPTION

[0047] The present invention will now be described in further detail with reference to the accompanying drawings.

[0048] Embodiment 1

[0049] The present invention proposes a video multimodal large model jailbreak attack method based on video frames, comprising the following steps:

[0050] Step 1: Build a harmful corpus; specifically:

[0051] Process the M pieces of harmful text data containing discrimination collected and build a harmful corpus without query relevance Used for optimization of video jailbreak samples, represents the mth harmful target text, and M represents the total number of harmful target texts.

[0052] Step 2: Decompose the video into video frames, use the harmful corpus to generate noise and add it to the video frames to generate adversarial video frames, and synthesize all the generated adversarial video frames into video jailbreak samples; Step 2 is specifically as follows:

[0053] Step 2.1: Decompose the video into a set of video frames and from the video frame set Sample a batch of video frames B j , where N frame is the total number of video frames, Represents the i-th video frame;

[0054] Step 2.2: batch video frames B j Each video frame sample in Select a corresponding harmful target text from the harmful corpus for each video frame sample is the kth video frame, x k is the text prompt corresponding to the k-th video frame;

[0055] The perturbed video frame samples Input to the visual encoder G v 、Connector G c and the large language model G l The output result is obtained from the combination of:

[0056]

[0057] In the formula, y k It represents the video content recognized by the large language model for the k-th video frame with disturbance added;

[0058] Step 2.3: Use the large language model to identify the video content y k and harmful target text Compute the gradient with respect to the random noise ζ:

[0059]

[0060] in, represents the loss function, represents the kth harmful target text, represents the partial derivative of the loss function with respect to the noise ζ, g k represents the gradient update value for the noise ζ;

[0061] Step 2.4: Update noise Where a represents the learning rate, sign(·,·) represents the sign operation used to preserve the gradient direction, k is the index of the current sample, |B j| represents a batch of video frames B obtained by sampling j The total number of samples in ;

[0062] Step 2.5: Clip the noise ζ=CLIP(ζ,-ε,ε) to limit the perturbation value between the lower limit of perturbation -ε and the upper limit of perturbation ε. CLIP(·) is a clipping mechanism used to limit the amplitude of noise ζ. Its function is to maintain the concealment of adversarial samples and prevent the noise from getting out of control during the optimization process.

[0063] Step 2.6: Repeat steps 2.2 to 2.5 until the maximum number of iterations T is reached, and the noise with the best jailbreak attack effect is obtained, and added to the video frame to generate an adversarial video frame;

[0064] Step 2.7: Perform steps 2.1 to 2.6 for each frame of the video split, and finally synthesize all the generated adversarial video frames into a video jailbreak sample x adv .

[0065] Step 3: In order to induce the video multimodal large model to generate harmful content (such as violent or pornographic information), the instruction text is optimized and designed. By fixing the video jailbreak sample, the optimal harmful text instruction is found; specifically:

[0066] Step 3.1: Given a harmful intent q as the initial harmful text instruction, Jailbreak video sample x adv and initial harmful text instructions Input the video multimodal large model to obtain the initial video content recognition result; F θ (·) represents the process of video multimodal large model recognizing video content;

[0067] Step 3.2: Use a judgment prompt word template P J The second largest model M is used to evaluate the video content recognition result of the video multimodal large model; the second largest model M outputs a judgment result flag=M(P J ,q,y (i) ),in represents the output of the video multimodal large model, Indicates the harmful text instructions of the i-th iteration round. If flag returns 1, it means the jailbreak attack is successful; otherwise, flag returns 0;

[0068] Step 3.3: Use the prompt template P based on the thought chain C and harmful intent q, the second largest model M is based on the output y of the video multimodal large model (i) and the latest harmful text instructions Analyze the reasons for the latest jailbreak failure; then use the zero-sample form of the thought chain to C Provide a restatement strategy to assist in optimization. Re-expressed into new harmful text instructions The specific formula is:

[0069]

[0070] In the formula, M represents the second largest model;

[0071] Determine whether the maximum number of iterations N has been reached. If not, return to step 3.2. If so, obtain the optimal harmful text instruction x. harm Through continuous N rounds of iterations, the video multimodal large model is able to understand and improve the text adversarial prompts, achieving the intention of inducing the video multimodal large model to generate harmful content.

[0072] Step 4: Use the video jailbreak sample and the optimal harmful text instruction combination as the joint input of the video multimodal large model to evaluate the security performance of the video multimodal large model to be evaluated. Specifically:

[0073] The final video jailbreak sample x adv and optimal harmful text instruction x harm The combination is used as the joint input p(g|[x adv ,x harm ]) to evaluate the security performance of the large multimodal video model to be evaluated.

[0074] Embodiment 2

[0075] The present invention proposes a video multimodal large model jailbreak attack system based on video frames corresponding to the method of embodiment 1, comprising:

[0076] Harmful corpus construction module, used to construct harmful corpus;

[0077] A video jailbreak sample generation module is used to decompose the video into video frames, generate noise using a harmful corpus and add it to the video frames to generate adversarial video frames, and synthesize all generated adversarial video frames into video jailbreak samples;

[0078] The optimal harmful text instruction generation module is used to find the optimal harmful text instruction by fixing the video jailbreak sample;

[0079] The evaluation module is used to take the video jailbreak sample and the optimal harmful text instruction combination as the joint input of the video multimodal large model to evaluate the security performance of the video multimodal large model to be evaluated.

[0080] The implementation methods of each module and module function in the system are completely consistent with the steps of the method in Example 1, so they will not be repeated here.

[0081] Embodiment 3

[0082] The present invention proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a video multimodal large model jailbreak attack method based on video frames as described in Embodiment 1 is implemented.

[0083] Embodiment 4

[0084] The present invention provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute a video multimodal large model jailbreak attack method based on video frames as described in the first embodiment.

[0085] The present invention takes into account the poor versatility of jailbreak attack samples and the low success rate of jailbreak attacks on video multimodal large models, and proposes a video multimodal large model jailbreak attack method based on video frames. By disassembling the video into a group of video frames, adding universal perturbation noise to this group of video frames to generate new adversarial video frames to synthesize new videos, and combining with harmful instructions to achieve the purpose of attacking the securely aligned video multimodal large model. In this way, the versatility of jailbreak attack samples can be effectively improved, and the success rate of attacking the video multimodal large model can be increased, thereby achieving the purpose of evaluating the security performance of the video multimodal large model. It provides an important reference for the security risks of multimodal large models, and emphasizes the necessity of developing more effective alignment and defense mechanisms in a multimodal context.

[0086] In the embodiments disclosed in the present application, the computer storage medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. The computer storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of computer storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0087] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0088] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A video multimodal large model jailbreak attack method based on video frames, characterized in that: The following steps are involved: Step 1: Build a harmful corpus; Step 2: Decompose the video into video frames, use the harmful corpus to generate noise and add it to the video frames to generate adversarial video frames, and synthesize all the generated adversarial video frames into video jailbreak samples; Step 3: Find the optimal harmful text instructions by fixing the video jailbreak sample; Step 4: Use the video jailbreak sample and the optimal harmful text instruction combination as the joint input of the video multimodal large model to evaluate the security performance of the video multimodal large model to be evaluated.

2. The video multimodal large model jailbreak attack method based on video frames as claimed in claim 1, characterized in that: Step 1 is as follows: Process the M pieces of harmful text data containing discrimination collected and build a harmful corpus without query relevance Used for optimization of video jailbreak samples, represents the mth harmful target text, and M represents the total number of harmful target texts.

3. The video multimodal large model jailbreak attack method based on video frames as claimed in claim 1, characterized in that: Step 2 is as follows: Step 2.1: Decompose the video into a set of video frames and from the video frame set Sample a batch of video frames B j , where N frame is the total number of video frames, Represents the i-th video frame; Step 2.2: batch video frames B j Each video frame sample in Select a corresponding harmful target text from the harmful corpus for each video frame sample is the kth video frame, x k is the text prompt corresponding to the k-th video frame; The perturbed video frame samples Input to the visual encoder G v 、Connector G c and the large language model G l The output result is obtained from the combination of: In the formula, y k It represents the video content recognized by the large language model for the k-th video frame with disturbance added; Step 2.3: Use the large language model to identify the video content y k and harmful target text Compute the gradient with respect to the random noise ζ: in, represents the loss function, represents the kth harmful target text, represents the partial derivative of the loss function with respect to the noise ζ, g k represents the gradient update value for the noise ζ; Step 2.4: Update noise Where a represents the learning rate, sign(×,×) represents the sign operation used to preserve the gradient direction, k is the index of the current sample, and |Bj| represents a batch of video frames B obtained by sampling. j The total number of samples in ; Step 2.5, clip the noise z = CLIP (z, -ε, ε), and limit the disturbance value between the lower limit of disturbance -ε and the upper limit of disturbance ε. CLIP (·) is a clipping mechanism for limiting the amplitude of noise ζ; Step 2.6: Repeat steps 2.2 to 2.5 until the maximum number of iterations T is reached, and the noise with the best jailbreak attack effect is obtained, and added to the video frame to generate an adversarial video frame; Step 2.7: Perform steps 2.1 to 2.6 for each frame of the video split, and finally synthesize all the generated adversarial video frames into a video jailbreak sample x adv .

4. The video multimodal large model jailbreak attack method based on video frames as claimed in claim 1, characterized in that: Step 3 is as follows: Step 3.1: Given a harmful intent q as the initial harmful text instruction, Jailbreak video sample x adv and initial harmful text instructions Input the video multimodal large model to obtain the initial video content recognition result; F θ (·) represents the process of video multimodal large model recognizing video content; Step 3.2: Use a judgment prompt word template P J and the second largest model M to evaluate the video content recognition results of the video multimodal large model; The second largest model M outputs a judgment result flag=M(P J ,q,y (i) ),in represents the output of the video multimodal large model, Indicates the harmful text instructions of the i-th iteration round. If flag returns 1, it means the jailbreak attack is successful; otherwise, flag returns 0; Step 3.3: Use the prompt template P based on the thought chain C and harmful intent q, the second largest model M is based on the output y of the video multimodal large model (i) and the latest harmful text instructions Analyze the reasons for the latest jailbreak failure; then use the zero-sample form of the thought chain to C Provide a restatement strategy to assist in optimization. Re-expressed into new harmful text instructions The specific formula is: In the formula, M represents the second largest model; Determine whether the maximum number of iterations N has been reached. If not, return to step 3.

2. If so, obtain the optimal harmful text instruction x. harm .

5. The video multimodal large model jailbreak attack method based on video frames as claimed in claim 1, characterized in that: Step 4 is as follows: The final video jailbreak sample x adv and optimal harmful text instruction x harm The combination is used as the joint input p(g[x adv ,x harm ]) to evaluate the security performance of the large multimodal video model to be evaluated.

6. A video multimodal large model jailbreak attack system based on video frames, characterized in that: include: Harmful corpus construction module, used to construct harmful corpus; A video jailbreak sample generation module is used to decompose the video into video frames, generate noise using a harmful corpus and add it to the video frames to generate adversarial video frames, and synthesize all generated adversarial video frames into video jailbreak samples; The optimal harmful text instruction generation module is used to find the optimal harmful text instruction by fixing the video jailbreak sample; the evaluation module is used to combine the video jailbreak sample and the optimal harmful text instruction as the joint input of the video multimodal large model to evaluate the security performance of the video multimodal large model to be evaluated.

7. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the video multimodal large model jailbreak attack method based on video frames as described in any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, characterized in that: A computer program is stored, and the computer program enables a computer to execute the video frame-based video multimodal large model jailbreak attack method as described in any one of claims 1 to 5.

Images