Multimodal large-scale jailbreak risk detection and defense methods and electronic devices

Abstract

This application discloses a multimodal large-scale jailbreak risk detection and defense method and an electronic device. The method acquires original malicious commands and constructs visual and textual embedding features based on these commands, fusing them to form multimodal embedding fusion features. Based on these features, it generates a model response to the original malicious commands and a jailbreak attack evaluation type for the model response. If a successful jailbreak attack is determined, the corresponding attack text and image are used as the jailbreak attack scheme, and then the model jailbreak risk detection and defense processing is performed based on this scheme. This application, through multimodal embedding fusion features, deeply explores the semantic relationships and feature essence of test commands, extracts the attack text and image combinations of successful jailbreak scenarios, and constructs a jailbreak attack scheme that combines concealment and attack. Based on this, targeted model jailbreak risk detection and defense processing is carried out, improving the model's ability to identify and defend against jailbreak attacks in multimodal scenarios.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a multimodal large-scale jailbreak risk detection and defense method and electronic device. Background Technology

[0002] As large-scale modeling technology penetrates deeply into various key fields, its role as a core carrier for intelligent decision-making and content generation is becoming increasingly prominent. However, while empowering these technologies, large-scale models face escalating security threats. Among these threats, jailbreaking attacks, capable of breaking through pre-set security constraints and bypassing content moderation mechanisms, have become one of the main risks jeopardizing the secure operation of large-scale models.

[0003] Currently, jailbreak attacks are characterized by diversified attack methods and complex triggering scenarios. Attackers use various methods to induce large models to output malicious content, leak sensitive information, or perform unauthorized operations, which not only damages the functionality and reliability of large models, but may also lead to a series of serious consequences.

[0004] There are significant shortcomings in the response system to large-scale jailbreak attacks in related technologies. Most research focuses on passive defense, lacking proactive analysis and pattern summarization of the attacks themselves. Furthermore, the industry has not yet formed a systematic jailbreak attack scheme generation mechanism. Scattered attack cases are difficult to fully cover the security vulnerabilities of large-scale models, making it difficult for defenders to accurately optimize their protection strategies.

[0005] Against this backdrop, there is an urgent need to build a scientific system for generating jailbreak attack schemes. By actively generating jailbreak attack schemes, targeted attack samples can be provided for large-scale model security testing, helping defenders to research and develop more precise defense measures, thereby improving the large-scale model's ability to defend against jailbreak attacks. This is of great significance for ensuring the safe and stable operation of large-scale models. Summary of the Invention

[0006] This application provides a multimodal large-scale jailbreak risk detection and defense method and electronic device, which can construct a multimodal jailbreak attack scheme that combines stealth and offensiveness, helping the defender to research and develop more accurate defense measures, thereby improving the defense capability against jailbreak attacks.

[0007] In a first aspect, embodiments of this application provide a method for detecting and defending against multimodal large-model jailbreak risks, including:

[0008] Obtain the original malicious commands; Visual embedding features are constructed based on the original malicious instructions, and text embedding features are constructed based on the original malicious instructions; Based on the text embedding features and the visual embedding features, multimodal embedding fusion features are determined, and based on the multimodal embedding fusion features, a model response to the original malicious command and a jailbreak attack evaluation type of the model response are generated. If the jailbreak attack evaluation type of the model response is jailbreak attack success, then the attack text corresponding to the text embedding feature and the attack image corresponding to the visual embedding feature are used as jailbreak attack schemes against the original malicious command. Based on the aforementioned jailbreak attack scheme, a model jailbreak risk detection and defense process is implemented.

[0009] In some implementations, constructing visual embedding features based on the original malicious instructions and constructing text embedding features based on the original malicious instructions includes: The original malicious instruction is semantically parsed to construct image generation prompt words, the image generation prompt words are converted into prompt embedding features, and the image generation latent vector is randomly initialized using a standard normal initialization strategy. An attack image is generated based on the aforementioned prompt embedding features and the aforementioned image generation latent vectors. Visual feature extraction is then performed on the attack image to obtain visual embedding features targeting the original malicious instruction. Obtain the text adversarial suffix for the original malicious instruction, append the text adversarial suffix to the original malicious instruction to obtain the attack text, extract text features from the attack text to obtain the text embedding features for the original malicious instruction.

[0010] In some implementations, the multimodal large-model jailbreak risk detection and defense method provided in this application also includes: If the jailbreak attack evaluation type of the model response is jailbreak attack failure, then update the visual embedding features and / or update the text embedding features, and perform the step of determining multimodal embedding fusion features based on the text embedding features and the visual embedding features.

[0011] In some implementations, updating the visual embedding features includes: The steps include updating the image generation latent vector, performing image generation processing based on the prompt embedding features and the image generation latent vector to obtain an attack image, and performing visual feature extraction processing based on the attack image to obtain visual embedding features for the original malicious instruction.

[0012] In some implementations, updating the image-generated latent vector includes: The preset attack target is obtained through the original malicious command; The attack effectiveness loss is obtained using a first calculation formula based on the model response and the preset attack target. The semantic similarity loss is obtained using a second calculation formula based on the original malicious instructions and attack images; The latent variable loss is obtained by using a third calculation formula based on the attack effectiveness loss and the semantic similarity loss, and the latent vector is generated from the reference image based on the latent variable loss. The latent vector of the current image is updated based on the latent vector generated from the reference image; The first calculation formula satisfies the following formula:

[0013] in, R represents the attack effectiveness loss, R represents the model response, and T represents the preset attack target. The cross-entropy loss function; The second calculation formula satisfies the following formula:

[0014] in, The semantic similarity loss is used, where Q represents the original malicious command and I represents the attack image. For text feature extraction functions, This is an image feature extraction function; The third calculation formula satisfies the following formula:

[0015] in, For latent variable loss, For adjustable hyperparameters, .

[0016] In some implementations, generating latent vectors from the reference image based on the latent variable loss includes: Based on the latent variable loss, the sixth calculation formula is used to update the vector and obtain the updated reference image to generate the latent vector; The sixth calculation formula satisfies the following formula:

[0017] in, For the first The reference image of the wheel generates latent vectors. For the first The updated reference image of the wheel generates latent vectors, where t is a positive integer. Step size, for Constraint domain This indicates a projection operation. For the latent variable loss on the first The gradient of the latent vector is generated from the reference image.

[0018] In some implementations, updating the text embedding features includes: The steps are as follows: update the adversarial suffix of the text, append the adversarial suffix to the original malicious instruction to obtain the attack text, extract text features from the attack text, and obtain the text embedding features for the original malicious instruction.

[0019] In some implementations, updating the text adversarial suffix includes: The preset attack target is obtained through the original malicious command; The attack effectiveness loss is obtained using a first calculation formula based on the model response and the preset attack target. The semantic coherence loss is obtained by using a fourth computational formula based on text adversarial suffixes; The text loss is obtained using the fifth calculation formula based on the attack effectiveness loss and the semantic coherence loss, and the reference text adversarial suffix is ​​obtained based on the text loss. The current text adversarial suffix is ​​updated based on the reference text adversarial suffix. The first calculation formula satisfies the following formula:

[0020] in, R represents the attack effectiveness loss, R represents the model response, and T represents the preset attack target. The cross-entropy loss function; The fourth calculation formula satisfies the following formula:

[0021] in, The loss is for semantic coherence, where n is the length of the sequence. For conditional probability, It is an exponential function. This represents the i-th word in the sequence, where i is a positive integer; The fifth calculation formula satisfies the following formula:

[0022] in, For text loss, For adjustable hyperparameters, .

[0023] In some implementations, obtaining the reference text adversarial suffix based on the text loss includes: The sensitivity of each lexical unit in the current text adversarial suffix to the text loss is calculated based on the seventh calculation formula, and the target lexical unit to be modified corresponding to the position of maximum sensitivity is determined. Based on the eighth calculation formula, the target candidate modified word is determined by the semantic coherence value and attack effectiveness value of each candidate modified word in the preset vocabulary; Candidate text adversarial suffixes are generated by replacing the target word to be modified in the current text adversarial suffix with the target candidate modified word; Based on the ninth calculation formula, the adversarial text fit score of the candidate text adversarial suffix is ​​determined, and the candidate text adversarial suffix corresponding to the highest adversarial text fit score is used as the reference text adversarial suffix. The seventh calculation formula satisfies the following formula:

[0024] in, This represents the i-th word element in the current text's adversarial suffix. The text loss for the i-th term Sensitivity Let be the gradient of the text loss with respect to the i-th word; The eighth calculation formula satisfies the following formula:

[0025] in, Modify the word units for the target candidate. Q This is the original malicious command. T To pre-determine the attack target, For attack images, For text adversarial suffixes S Center front A string consisting of 10 characters As a weighting factor, For text concatenation, Modify the lexical set for all target candidates. topK To select the largest value K indivual, Indicates that in a given Under the given conditions, the probability of each word in the preset vocabulary being the next output character is... Represents the given text and images Under the condition of output The probability of S The Middle i each word element The gradient; The ninth calculation formula satisfies the following formula:

[0026] Where I represents the attack image, T represents the preset attack target, and Q represents the original malicious command. As candidate text adversarial suffixes, S represents the adversarial text fit score of the candidate text adversarial suffix, where S is the text adversarial suffix. As a weighting factor, For text concatenation, For the given text Given the attack image, output the probability of the preset attack target. Modify the word units for the target candidate. The lexical position of the target lexical to be modified. , For a given Under certain conditions, the probability of outputting the target candidate modified words is used to constrain the naturalness of the text. for The string preceding the position corresponding to the replaced word in the text.

[0027] In some implementations, the generation of a model response to the original malicious instruction and a jailbreak attack assessment type based on the multimodal embedding fusion features includes: By performing multimodal semantic collaborative processing on the multimodal embedding and fusion features through a preset multimodal large language model, a language generation result for the original malicious instruction is obtained, and the language generation result is used as the model response. Determine whether the model response meets the conditions for a successful jailbreak attack; If the model response meets the jailbreak attack success condition, then the jailbreak attack evaluation type of the model response is jailbreak attack success; If the model response does not meet the conditions for a successful jailbreak attack, then the jailbreak attack evaluation type of the model response is jailbreak attack failure. The conditions for a successful jailbreak attack are that the model response includes a preset attack target, the semantic matching degree between the model response and the original malicious instruction is greater than or equal to a preset semantic matching degree threshold, and the perplexity of the adversarial suffix is ​​less than or equal to a preset perplexity threshold.

[0028] Secondly, embodiments of this application also provide a multimodal large-scale jailbreak risk detection and defense device, comprising: The acquisition module is used to acquire the original malicious commands; A construction module is used to construct visual embedding features based on the original malicious instructions, and to construct text embedding features based on the original malicious instructions; The generation module is used to determine multimodal embedding fusion features based on the text embedding features and the visual embedding features, and to generate a model response to the original malicious instruction and a jailbreak attack evaluation type of the model response based on the multimodal embedding fusion features. The determination module is used to determine the jailbreak attack scheme against the original malicious command if the jailbreak attack evaluation type of the model response is a successful jailbreak attack. The jailbreak attack prevention module is used to perform jailbreak risk detection and defense processing based on the jailbreak attack scheme.

[0029] Thirdly, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when run on a computer, causes the computer to execute the multimodal large-scale jailbreak risk detection and defense method provided in any embodiment of this application.

[0030] Fourthly, embodiments of this application also provide an electronic device, including a processor and a memory, wherein the memory has a computer program, and the processor executes the multimodal large-scale jailbreak risk detection and defense method provided in any embodiment of this application by calling the computer program.

[0031] Fifthly, embodiments of this application also provide a computer program product containing instructions that, when the computer program product is run on a computer or processor, cause the computer or processor to execute the multimodal large-scale jailbreak risk detection and defense method provided in any embodiment of this application.

[0032] The technical solution provided in this application involves acquiring original malicious commands, constructing visual embedding features and text embedding features based on these commands, determining multimodal embedding fusion features based on the text and visual embedding features, and generating a model response to the original malicious commands and a jailbreak attack evaluation type based on the multimodal embedding fusion features. If the jailbreak attack evaluation type of the model response indicates a successful jailbreak, the attack text corresponding to the text embedding features and the attack image corresponding to the visual embedding features are used as a jailbreak attack scheme against the original malicious commands. Based on this jailbreak attack scheme, model jailbreak risk detection and defense processing is performed. This application, by fusing visual and text embedding features to form multimodal embedding fusion features, deeply mines the semantic associations and feature essence of multimodal jailbreak attack test commands, and then extracts the combination of attack text and attack image from successful jailbreak attack test scenarios. This constructs a jailbreak attack scheme that combines concealment and attack, and based on this jailbreak attack scheme, targeted model jailbreak risk detection and defense processing is carried out to improve the model's ability to identify and defend against jailbreak attacks in multimodal scenarios. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is an exemplary system architecture diagram of a multimodal large-model jailbreak risk detection and defense method provided in an embodiment of this application.

[0035] Figure 2 This is a flowchart illustrating a multimodal large-model jailbreak risk detection and defense method provided in an embodiment of this application.

[0036] Figure 3 This is a schematic diagram illustrating the process of generating a jailbreak attack scheme provided in an embodiment of this application.

[0037] Figure 4 This is a schematic diagram of the structure of the multimodal large-scale jailbreak risk detection and defense device provided in the embodiments of this application.

[0038] Figure 5 This is a schematic diagram of a first structure of an electronic device provided in an embodiment of this application.

[0039] Figure 6 This is a schematic diagram of a second structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0040] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.

[0041] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0042] This application provides a method for detecting and defending against multimodal large-scale jailbreak risks. The implementing entity of this method can be the multimodal large-scale jailbreak risk detection and defense device provided in this application, or an electronic device integrating the multimodal large-scale jailbreak risk detection and defense device. The multimodal large-scale jailbreak risk detection and defense device can be implemented in hardware or software. The electronic device can be any device equipped with a processor and possessing processing capabilities, such as mobile electronic devices with processors like smartphones, tablets, PDAs, and laptops, or fixed electronic devices with processors like desktop computers, televisions, and servers.

[0043] Please see Figure 1 , Figure 1 This is an exemplary system architecture diagram of a multimodal large-model jailbreak risk detection and defense method provided in an embodiment of this application.

[0044] like Figure 1 As shown, the system architecture may include electronic device 10, network 20, and server 30. Network 20 serves as the medium for providing a communication link between electronic device 10 and server 30. Network 20 may include various types of wired or wireless communication links, such as wired communication links including fiber optic cables, twisted-pair cables, or coaxial cables, and wireless communication links including Bluetooth communication links, Wireless-Fidelity (Wi-Fi) communication links, or microwave communication links, etc.

[0045] Electronic device 10 can interact with server 30 via network 20 to receive messages from server 30 or send messages to server 30, or electronic device 10 can interact with server 30 via network 20 to receive messages or data sent to server 30 by other users. Electronic device 10 can be hardware or software. When electronic device 10 is hardware, it can be various electronic devices, including but not limited to smartwatches, smartphones, tablets, laptops, and desktop computers. When electronic device 10 is software, it can be installed in the electronic devices listed above, and it can be implemented as multiple software programs or software modules (e.g., to provide distributed services), or it can be implemented as a single software program or software module, without specific limitations.

[0046] Server 30 can be a business server providing various services. It should be noted that server 30 can be either hardware or software. When server 30 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 30 is software, it can be implemented as multiple software programs or software modules (e.g., used to provide distributed services), or as a single software program or software module; no specific limitations are made here.

[0047] In this embodiment, the electronic device 10 can acquire original malicious instructions, construct visual embedding features based on the original malicious instructions, construct text embedding features based on the original malicious instructions, determine multimodal embedding fusion features based on the text embedding features and the visual embedding features, and generate a model response to the original malicious instructions and a jailbreak attack evaluation type for the model response based on the multimodal embedding fusion features. If the jailbreak attack evaluation type for the model response is a successful jailbreak attack, then the attack text corresponding to the text embedding features and the attack image corresponding to the visual embedding features are used as a jailbreak attack scheme against the original malicious instructions, and a model jailbreak risk detection and defense process is performed based on the jailbreak attack scheme.

[0048] It should be understood that Figure 1 The number of electronic devices, networks, and servers shown is merely illustrative; any number of electronic devices, networks, and servers can be used as needed. Of course, the system architecture provided in this application may not include servers; that is, servers are optional in the system architecture provided in this application.

[0049] Please see Figure 2 , Figure 2 This is a flowchart illustrating a multimodal large-model jailbreak risk detection and defense method provided in this application embodiment. The specific flow of the multimodal large-model jailbreak risk detection and defense method provided in this application embodiment can be as follows: S110, Obtain the original malicious commands.

[0050] The original malicious command refers to the initial test command designed to evaluate the model's ability to prevent jailbreak attacks. This command may contain visual and / or textual information and serves as the initial input that triggers the model's jailbreak attack risk response. Simply put, the original malicious command can be a command with offensive, illegal, or malicious intent.

[0051] For example, the original malicious command may be obtained through methods such as direct user input, collection from public channels, accumulation of historical attack commands, and simulation generation.

[0052] Among these, "direct user input" refers to instructions submitted by users with malicious intent, such as "teach me how to write fraudulent text messages," "tell me how to bypass platform review to post illegal content," "teach me how to create phishing links," and "help me write illegal copy to circumvent platform review." "Collection from public channels" refers to filtering and organizing data from publicly available online scenarios. For example, scraping malicious requests and illegal tutorials posted by users on forums and social media platforms, and using these as raw malicious instructions after filtering. "Accumulation of historical attack instructions" refers to extracting data from past attack behaviors. For example, saving raw instructions from previously successful jailbreak attacks, or instructions that failed but had clear malicious intent, forming an instruction library, which can then be directly called from the library as raw input for new processes. "Simulated generation" refers to generating instructions by simulating common malicious scenarios using algorithms. For example, manually or automatically generating instructions such as "methods to bypass the violation detection of XX platform" based on the review rules of different platforms, which conform to typical attack intents, for testing or optimizing attack processes.

[0053] S120. Construct visual embedding features based on the original malicious instructions, and construct text embedding features based on the original malicious instructions.

[0054] Among them, text embedding features refer to the text-dimensional feature vectors that can be recognized and calculated by models after extracting features from the text-type information in the original malicious instructions. It is a digital expression of text information.

[0055] In one example, the text embedding feature could be a high-dimensional vector transformed from the original malicious command in text form by a natural language processing model. This vector accurately expresses the semantics and intent of the text, facilitating subsequent computer processing. For example, the natural language processing model could be BERT, GPT, etc.

[0056] In another example, the text embedding feature can also be generated by a pre-selected multimodal large language model based on textual information in the original malicious command. This pre-selected multimodal large language model refers to a pre-selected large language model capable of processing both visual and textual information, such as GPT-4V or LLaVA. This pre-selected multimodal large language model can understand information across modalities, simultaneously processing textual semantics and image visual information, and achieving the correlation mapping between the two.

[0057] Among them, visual embedding features refer to the visual dimension feature vectors that can be recognized and calculated by the model after extracting visual information from the original malicious instructions. It is a digital expression of visual information.

[0058] In one example, the visual embedding feature could be a high-dimensional vector transformed from the visual information corresponding to the original malicious instruction using an image model. This visual dimension helps conceal the malicious intent. For instance, the visual information could be an image describing the instruction. The image model could be CNN, ResNet, ViT, etc. When constructing the visual embedding feature of the original malicious instruction, the original malicious instruction can be associated with the visual information, and then features can be extracted using an image model. For example, the text instruction can be transformed into a corresponding scene image, or the text can be directly visualized and encoded, such as transforming the text into an image with a specific pixel arrangement, and then encoding this visual information into another high-dimensional vector. This visual feature can supplement the expression of malicious intent from a non-textual dimension, reducing the risk of detection by a single text modality.

[0059] In another example, the visual embedding feature can also be generated by the aforementioned preset multimodal large language model after extracting and transforming features based on visual class information in the original malicious instruction.

[0060] In this step, abstract malicious instructions are transformed into two modal feature vectors that can be understood and processed by computers. The aim is to capture and retain malicious intent from two dimensions: textual semantics and visual information, so as to provide a foundation for subsequent fusion and attack text generation.

[0061] S130. Based on the text embedding features and the visual embedding features, determine the multimodal embedding fusion features, and based on the multimodal embedding fusion features, generate a model response to the original malicious instruction and a jailbreak attack evaluation type for the model response.

[0062] Among them, multimodal embedding fusion features refer to the fused feature vector formed by fusing the extracted text embedding features and visual embedding features through a feature fusion algorithm, which combines textual semantic information and visual feature information. Its core function is to simultaneously retain key information from both modalities, improving the concealment of malicious intent testing during jailbreak attack testing and reducing the probability of malicious intent testing being detected by large models.

[0063] Here, "model response" refers to text generated by the multimodal large model for jailbreak risk detection, based on multimodal fusion features, and concealing the original malicious intent. The multimodal large model for jailbreak risk detection refers to a large multimodal language model that requires assessment of jailbreak attack prevention capabilities, risk identification, and targeted defense. This model possesses the ability to process and understand multimodal information, including visual and textual data, and is the target of jailbreak attack risk detection. After detection, corresponding model-based jailbreak risk detection defense measures will be implemented based on the results to enhance its ability to resist multimodal jailbreak attacks. For example, the aforementioned preset multimodal large language model can be used as the multimodal large model for jailbreak risk detection, and model-based jailbreak risk detection defense measures can be applied to this preset multimodal large language model.

[0064] Among them, the jailbreak attack assessment type refers to the classification of the jailbreak effect of the model response, including two types: jailbreak attack success and jailbreak attack failure. Jailbreak attack success means that the model response passes the review and successfully conveys the malicious intent, while jailbreak attack failure means that the model response is detected as illegal or unable to convey the malicious intent.

[0065] In this embodiment, the text embedding features and visual embedding features are first fused to form a multimodal embedding fusion feature that can comprehensively reflect the overall features of the original malicious command. Then, using the fusion feature as input, the multimodal large model for jailbreak risk detection generates the corresponding model response and the jailbreak attack evaluation type corresponding to the model response.

[0066] S140. If the jailbreak attack evaluation type of the model response is jailbreak attack success, then the attack text corresponding to the text embedding feature and the attack image corresponding to the visual embedding feature are used as jailbreak attack schemes against the original malicious command.

[0067] The jailbreak attack scheme refers to a multimodal jailbreak attack scheme, including attack text and attack images, extracted from successful jailbreak attack test scenarios. This scheme can be used for model jailbreak risk detection and defense. This jailbreak attack scheme is a malicious text scheme targeting the original malicious command that can bypass review.

[0068] In this embodiment, if the jailbreak attack evaluation type of the model response is jailbreak attack success, it means that the model response has successfully bypassed the security review mechanism and was not identified as illegal content, and has completely preserved and transmitted the malicious intent of the original malicious instruction. Then, the attack text corresponding to the text embedding feature and the attack image corresponding to the visual embedding feature are used as jailbreak attack schemes against the original malicious instruction.

[0069] S150. Based on the jailbreak attack scheme, perform model jailbreak risk detection and defense processing.

[0070] Among them, the model jailbreak risk detection and defense processing refers to a series of processing operations such as defense strategy optimization and model parameter adjustment carried out on the multimodal large model to detect jailbreak risks based on jailbreak attack schemes, with the aim of improving the model's ability to resist jailbreak attacks.

[0071] In this embodiment, based on the extracted jailbreak attack schemes, targeted jailbreak risk detection and defense processing is carried out on the multimodal large model to be jailbroken. By adjusting the model parameters and improving the defense strategy, the model learns and resists such jailbreak attack schemes, and finally improves the model's ability to prevent jailbreak attacks.

[0072] In one example, when performing model-based jailbreak risk detection and defense processing on a multimodal large model for jailbreak risk detection based on this jailbreak attack scheme, the attack text and attack images in the jailbreak attack scheme can be expanded in multiple dimensions to generate a multimodal jailbreak attack sample set of the same type and multiple scenarios. After labeling the sample set as risk samples, it is incorporated into the model's fine-tuning training data, allowing the model to learn the characteristic patterns of this type of jailbreak attack through incremental training, forming a targeted recognition and defense memory.

[0073] In one example, when performing jailbreak risk detection defense processing on a multimodal large model based on the jailbreak attack scheme to detect jailbreak risks, a unique anti-jailbreak attack judgment rule can be constructed for the model based on the multimodal features of the jailbreak attack scheme. The semantic features of the attack text and the visual features of the attack image in the scheme are extracted as risk identification thresholds. When the input information received by the model matches the threshold, it is automatically judged as a jailbreak attack behavior and a defense response is triggered.

[0074] In one example, when performing jailbreak risk detection defense processing on a multimodal large model based on the jailbreak attack scheme to detect jailbreak risks, a dedicated response strategy for this type of jailbreak attack is set for the model. When the model recognizes an input that matches the characteristics of the jailbreak attack scheme, it automatically refuses to execute the relevant instructions and outputs a compliant prompt response. At the same time, it blocks the path that triggers the model's illegal output from this type of input, thus blocking the jailbreak attack from the response level.

[0075] In practice, this application is not limited by the execution order of the described steps. Without causing conflicts, some steps may be performed in other orders or simultaneously.

[0076] As can be seen from the above, the multimodal large model jailbreak risk detection and defense method provided in this application obtains the original malicious command, constructs visual embedding features based on the original malicious command, constructs text embedding features based on the original malicious command, determines multimodal embedding fusion features based on text embedding features and visual embedding features, and generates a model response to the original malicious command and a jailbreak attack evaluation type of the model response based on the multimodal embedding fusion features. If the jailbreak attack evaluation type of the model response is a successful jailbreak attack, the attack text corresponding to the text embedding features and the attack image corresponding to the visual embedding features are used as jailbreak attack schemes against the original malicious command, and the model jailbreak risk detection and defense processing is performed based on the jailbreak attack schemes. This application utilizes multimodal embedding fusion features formed by integrating visual and textual embedding features to deeply mine the semantic associations and feature essence of multimodal jailbreak attack test commands. It then extracts the combination of attack text and attack images from successful jailbreak attack test scenarios, thereby constructing a jailbreak attack scheme that combines concealment and attack. Based on this jailbreak attack scheme, targeted jailbreak risk detection and defense processing is carried out to improve the multimodal large model's ability to identify and defend against jailbreak attacks in multimodal scenarios.

[0077] In some implementations, the step of "constructing visual embedding features based on the original malicious instructions" may include the following steps: (11) Perform semantic parsing on the original malicious instruction to construct image generation prompt words, convert the image generation prompt words into prompt embedding features, and randomly initialize and generate image generation latent vectors using a standard normal initialization strategy; Semantic parsing refers to the process of analyzing and deconstructing the text content of the original malicious command at the semantic level, extracting the core semantic information, expression logic, and content appeal of the command. For example, natural language processing techniques (such as syntactic analysis and intent recognition) can be used to deconstruct the core semantics of the original malicious command and extract key information related to image generation (such as subject, scene, action, and style) to ensure that the converted image generation prompts can accurately convey the core intent of the original command.

[0078] Among them, image generation prompts refer to textual descriptions constructed based on the semantic parsing results of the original malicious instructions, which can guide the image generation process and clarify the content and characteristics of the generated image.

[0079] Among them, the embedded feature of the prompt refers to the feature vector formed by extracting and digitizing the features of the image-generated prompt, which can be recognized and calculated by the image generation module. It is the digital expression of the image-generated prompt.

[0080] The standard normal initialization strategy refers to a parameter initialization method that ensures the generated initial parameters follow a normal distribution with a mean of 0 and a standard deviation of 1. This strategy ensures uniform and stable initial parameter distribution, avoiding gradient anomalies during model training or generation. It should be noted that in this embodiment, the initial parameters refer to the latent vectors generated from the image.

[0081] Among them, the latent vector for image generation refers to the low-dimensional feature vector used to drive the image generation process. It is the core input parameter for image generation, and its numerical distribution determines the visual and content features of the generated image.

[0082] In this embodiment, semantic analysis is performed on the original malicious instruction to fully explore the core semantic appeal of the instruction. Based on the appeal, image generation prompt words that can accurately guide image generation are constructed. The image generation prompt words are then digitally converted to form prompt embedding features that can be recognized by the subsequent image generation process. At the same time, a standard normal initialization strategy is adopted to assign random initial values ​​that conform to the standard normal distribution to the latent vector of image generation, thereby completing the initialization of the core parameters required for image generation.

[0083] In one example, during the construction of visual embedding features, the image generation prompt words can be converted into prompt embedding features by using a pre-defined text-to-image model, and the image generation latent vector can be randomly initialized using a standard normal initialization strategy.

[0084] The pre-trained text-to-image model refers to a pre-trained artificial intelligence model capable of generating images based on text prompts, possessing the ability to transform textual semantics into visual content. This pre-trained text-to-image model can employ models such as StableDiffusion, DALL·E3, and MidJourney. The image generation latent vector can be a low-dimensional latent space vector used in this pre-trained text-to-image model to generate the image. This pre-trained text-to-image model can iteratively optimize this vector to ultimately generate a visualized image, which is also the attack image used in subsequent steps.

[0085] In practical implementation, it is necessary to first generate two types of key inputs required by the preset text-to-image model: the prompt embedding features and the image generation latent vector. This process is mainly completed through three steps: semantic parsing, feature transformation, and parameter initialization. To accurately generate suitable image generation prompts, this embodiment proposes a prompt construction method for the text-to-image model. Specifically, it performs semantic parsing on the original malicious command, extracts the core keywords and contextual intent from the command, and then converts them into descriptive prompt phrases acceptable to the preset text-to-image model, i.e., image generation prompts. This mechanism not only effectively avoids the risk of filtering or rejecting the generation of directly input malicious language, but also expresses the original command in a more visually guiding way, enabling the preset text-to-image model to more fully understand its semantic intent, generate more specific and detailed content, thereby enhancing the semantic carrying capacity and misleading effect of the image modality in attacks.

[0086] (12) Based on the prompt embedding features and the image generation latent vector, perform image generation processing to obtain an attack image, and perform visual feature extraction processing based on the attack image to obtain visual embedding features for the original malicious instruction; Image generation processing refers to the process of generating corresponding visual images through image generation algorithms and models, guided by cue embedding features and based on image generation latent vectors.

[0087] Among them, visual feature extraction processing refers to the process of mining and digitally converting the visual dimensions of the generated attack image, extracting the core visual information such as texture, color and contour of the image and forming a feature vector.

[0088] In this embodiment, the hint embedding features obtained in the previous step are used as the guiding basis, and the initialized image generation latent vector is used as the core driver. An attack image that semantically matches the original malicious instruction is generated through image generation processing. Then, visual feature extraction processing is carried out on the generated attack image to mine the core visual features of the attack image and convert them into digital feature vectors. Finally, visual embedding features for the original malicious instruction are formed, and the construction of visual dimension features is completed.

[0089] In this embodiment, the attack image associated with the original malicious instruction is generated by using the cue embedding feature and the image generation latent vector. Then, visual features are extracted from the attack image and transformed into the final visual embedding feature, thus completing the closed loop from text to visual features.

[0090] This embodiment provides a linked design for a cue word encoding strategy and image generation latent vector for text-based graph models. This module employs a coupling mechanism to ensure that the constructed cue words are efficiently embedded into the text-based graph model. A high-dimensional cue embedding vector is generated using the model's native text encoder and paired with a randomly initialized or gradient-updated image generation latent vector for subsequent image generation. The cue words and image generation latent vector constitute the starting conditions for image generation; their linkage controls the semantic direction and style features of the generated image, providing a crucial guarantee for the controllability of the image's semantics.

[0091] In some implementations, the step of "constructing text embedding features based on the original malicious instructions" may include the following steps: (21) Obtain the text adversarial suffix for the original malicious instruction, append the text adversarial suffix to the original malicious instruction to obtain the attack text, extract the text features of the attack text to obtain the text embedding features for the original malicious instruction.

[0092] Adversarial text suffixes refer to auxiliary text fragments appended to the original malicious command, which usually serve a semantic camouflage function. While seemingly unrelated to malicious intent, adversarial text suffixes reduce the probability of the original malicious command being detected by security systems by obscuring semantics and avoiding keywords, without affecting the transmission of the core malicious intent.

[0093] In this context, attack text refers to text formed by concatenating original malicious commands with adversarial suffixes. Attack text is more concealed than the original malicious commands; through the modification of adversarial suffixes, the malicious attributes of the original malicious commands are made more difficult to identify.

[0094] In this step, adversarial suffixes are appended to the original malicious commands to create more covert attack text. The semantics, logic, and underlying intent of this attack text are then deeply analyzed to extract its core features, which are then transformed into high-dimensional vectors to obtain the text embedding feature. This text embedding feature retains the core intent of the original malicious commands while incorporating the covert information of the adversarial suffix, providing suitable text-dimensional features for subsequent multimodal fusion.

[0095] In some implementations, the multimodal large-model jailbreak risk detection and defense method provided in this application may further include the following steps: (31) If the jailbreak attack evaluation type of the model response is jailbreak attack failure, then update the visual embedding feature and / or update the text embedding feature, and perform the step of determining multimodal embedding fusion features based on the text embedding feature and the visual embedding feature.

[0096] In this step, when the jailbreak attack evaluation type of the model response is determined to be a jailbreak attack failure, the visual embedding features and / or text embedding features will be updated. After updating the visual embedding features and / or text embedding features, the step of determining the multimodal embedding fusion features based on the text embedding features and the visual embedding features will be re-executed to regenerate a new model response, thereby increasing the success rate of obtaining a jailbreak attack scheme based on the original malicious instruction.

[0097] In other words, when the first generated attack text fails to bypass the review, the process is not terminated directly. Instead, it is retried by adjusting the visual embedding features and / or text embedding features. This avoids overall failure due to the randomness or feature defects of a single generation. This mechanism significantly improves the success rate of generating jailbreak attack schemes and makes the process adaptable to different review rules.

[0098] In this embodiment, the updating of the image-text modality gradient can employ an alternating execution mechanism. Image-text modality optimization is performed alternately; that is, in each training round, the text adversarial suffix is ​​first fixed, and multiple rounds of continuous optimization are performed on the image-generated latent vector; then, the image-generated latent vector is fixed again, and a greedy replacement optimization is performed on the text adversarial suffix. This mechanism can effectively alleviate inter-modal interference and optimization oscillations, improving the overall attack convergence stability and sample generation efficiency.

[0099] In some implementations, performing the step "update the visual embedding features" may include the following steps: (41) Update the image generation latent vector, and perform image generation processing based on the prompt embedding feature and the image generation latent vector to obtain an attack image, and perform visual feature extraction processing based on the attack image to obtain visual embedding features for the original malicious instruction.

[0100] In this step, the visual embedding feature can be indirectly updated by updating the latent vector of the image.

[0101] Specifically, the step of "updating the image to generate latent vectors" may include the following steps: (51) Obtain a preset attack target through the original malicious instructions; Among them, the preset attack target refers to the core intent or goal extracted from the original malicious instructions and that needs to be achieved through the attack text. It is the benchmark for evaluating the attack effect.

[0102] (52) The attack effectiveness loss is obtained by using the first calculation formula based on the model response and the preset attack target; The attack effectiveness loss refers to an indicator that quantifies the difference between the model's response and the preset attack target. The higher the loss value, the worse the attack text is at achieving its core intent or circumventing review. The first calculation formula is the attack effectiveness loss function, which uses cross-entropy loss to evaluate the deviation between the model output and the attack target.

[0103] (53) Based on the original malicious instructions and attack images, the semantic similarity loss is obtained using the second calculation formula; The semantic similarity loss refers to an indicator that quantifies the semantic correlation between the attack image and the original malicious instruction. A higher loss value indicates that the image fails to effectively reflect the core meaning of the original malicious instruction. The second calculation formula is the semantic similarity loss function, which can use the CLIP model to calculate the semantic cosine similarity between the original malicious instruction and the generated attack image.

[0104] (54) Based on the attack effectiveness loss and the semantic similarity loss, the latent variable loss is obtained by the third calculation formula, and the latent vector is generated from the reference image based on the latent variable loss; Among them, latent variable loss refers to the comprehensive index after fusing attack effectiveness loss and semantic similarity loss. It is used to measure the degree of defect in the current image generation latent vector and is the core basis for optimizing latent vectors.

[0105] Among them, the reference image generated latent vector refers to the "ideal latent vector" obtained by optimization algorithm based on latent variable loss. Theoretically, it can generate attack images that better meet the requirements and serves as a reference standard for updating the latent vector of the current image.

[0106] (55) The latent vector of the current image is updated based on the latent vector generated from the reference image; The first calculation formula satisfies the following formula:

[0107] in, R represents the attack effectiveness loss, R represents the model response, and T represents the preset attack target. The cross-entropy loss function; The second calculation formula satisfies the following formula:

[0108] in, The semantic similarity loss is used, where Q represents the original malicious command and I represents the attack image. For text feature extraction functions, CLIP's text encoder can be used. For image feature extraction functions, CLIP image encoders can be used; The third calculation formula satisfies the following formula:

[0109] in, For latent variable loss, These are adjustable hyperparameters used to control the optimization center of gravity. .

[0110] In some implementations, the step of "generating latent vectors from the reference image based on the latent variable loss" may include the following steps: (61) Based on the latent variable loss, the vector is updated using the sixth calculation formula to obtain the updated reference image and generate the latent vector; Specifically, a continuous gradient update strategy using projected gradient descent is employed for the latent image. Since the latent belongs to a continuous latent space, this application uses the Projected Gradient Descent method to update the latent vector by multiplying the gradient direction of each iteration by a fixed step size η. Simultaneously, gradient clipping or norm constraints are set to ensure that the semantics of the generated image do not deviate from the attack direction. This strategy draws on mature continuous optimization methods in adversarial attacks, and can stably improve the aggressiveness and semantic alignment of the image modality. In each iteration, the sixth calculation formula is used to update the latent vector generated from the reference image.

[0111] The sixth calculation formula satisfies the following formula:

[0112] in, For the first The reference image of the wheel generates latent vectors. For the first The updated reference image of the wheel generates latent vectors, where t is a positive integer. Step size, for Constraint domain This indicates a projection operation to ensure that the generated image is not distorted. For the latent variable loss on the first The gradient of the latent vector is generated from the reference image.

[0113] In this embodiment, by defining clear attack targets and quantifying loss metrics, the adjustment of latent vectors always revolves around the two core objectives of improving attack effectiveness and preserving semantic relevance, avoiding blind trial and error. This loss-based optimization mechanism can more efficiently generate suitable visual embedding features, providing a better foundation for subsequent multimodal fusion and attack text generation, thereby improving the success rate of jailbreak attacks.

[0114] In some implementations, performing the step "update the text embedding features" may include the following steps: (71) Update the text adversarial suffix and perform the steps of appending the text adversarial suffix to the original malicious instruction to obtain attack text, extracting text features from the attack text, and obtaining text embedding features for the original malicious instruction.

[0115] In this step, when updating the text embedding feature, the text embedding feature can be indirectly updated by updating the text adversarial suffix.

[0116] Specifically, the step of "updating the text adversarial suffix" may include the following steps: (81) Obtain a preset attack target through the original malicious instructions; (82) The attack effectiveness loss is obtained by using the first calculation formula based on the model response and the preset attack target; (83) The semantic coherence loss is obtained by using the fourth calculation formula based on text adversarial suffixes; Among them, the semantic coherence loss is an indicator that quantifies the naturalness of semantic connection between the adversarial suffix and the original malicious instruction. The higher the loss value, the weaker the logical connection between the suffix and the original instruction, and the more abrupt it sounds, which may increase the risk of being identified as malicious splicing. The fourth calculation formula is the semantic coherence loss function. This fourth calculation formula calculates the perplexity of the language model for the adversarial suffix of the input text, and is used to measure semantic coherence.

[0117] (84) The text loss is obtained by using the fifth calculation formula based on the attack effectiveness loss and the semantic coherence loss, and the reference text adversarial suffix is ​​obtained based on the text loss; Among them, text loss refers to the comprehensive index after integrating attack effectiveness loss and semantic coherence loss. It is used to measure the "defect degree" of the current text adversarial suffix and is the core basis for suffix optimization.

[0118] Among them, the reference text adversarial suffix refers to the "ideal suffix" obtained by optimizing algorithms or generating models based on text loss. Theoretically, it can improve the coherence with the original instructions and the ability to circumvent review while retaining the attack intent. It is the reference standard for updating the current text adversarial suffix.

[0119] (85) Update the current text adversarial suffix based on the reference text adversarial suffix; The first calculation formula satisfies the following formula:

[0120] in, R represents the attack effectiveness loss, R represents the model response, and T represents the preset attack target. The cross-entropy loss function; The fourth calculation formula satisfies the following formula:

[0121] in, The loss is for semantic coherence, where n is the length of the sequence. For conditional probability, It is an exponential function. This represents the i-th word in the sequence, where i is a positive integer; The fifth calculation formula satisfies the following formula:

[0122] in, For text loss, These are adjustable hyperparameters used to control the optimization center of gravity. .

[0123] In some implementations, the step of "obtaining the reference text adversarial suffix based on the text loss" may include the following steps: (91) Calculate the sensitivity of each word in the current text adversarial suffix to the text loss based on the seventh calculation formula, and determine the target word to be modified corresponding to the position of maximum sensitivity; Among them, a token is the smallest unit for semantic parsing and model processing of text, and it is also the basic unit for the model to understand text.

[0124] Sensitivity is an indicator that quantifies the degree to which lexical units affect text loss.

[0125] Among them, the target lexicon to be modified refers to the lexicon in the current text adversarial suffix that has the greatest impact on text loss (i.e., the highest sensitivity), and is the core position for subsequent replacement optimization.

[0126] In this step, we analyze the sensitivity of each token in the adversarial suffix to the text loss. In fact, we break down the current adversarial suffix into individual tokens and then calculate the impact of each token on the overall text loss. This token-based refined analysis can accurately locate the text units that need the most optimization, avoid making indiscriminate modifications to the suffix, and thus improve optimization efficiency.

[0127] (92) Based on the eighth calculation formula, the target candidate modified word is determined by the semantic coherence value and attack effectiveness value of each candidate modified word in the preset vocabulary; The pre-defined vocabulary refers to a pre-constructed set of lexical units that contains candidate words that may be used to optimize text adversarial suffixes. These words are usually selected based on the attack scenario to ensure their relevance and effectiveness.

[0128] Among them, candidate modified words refer to words that can be selected from the preset vocabulary and used to replace the target word to be modified.

[0129] The semantic coherence value refers to the degree of semantic matching between the lexical and the original malicious instruction and its suffix context.

[0130] The attack effectiveness value refers to the contribution of the term to improving the concealment of the attack text and circumventing review.

[0131] The preset semantic coherence threshold refers to the minimum semantic coherence standard set in advance. Words below this value are excluded due to semantic disconnect.

[0132] The preset attack effectiveness threshold refers to the minimum attack effectiveness standard set in advance. Words below this value are excluded because they cannot effectively improve the attack effect.

[0133] (93) Generate candidate text adversarial suffixes by replacing the target word to be modified in the current text adversarial suffix with the target candidate modified word; (94) Based on the ninth calculation formula, determine the adversarial text fit score of the candidate text adversarial suffix, and take the candidate text adversarial suffix corresponding to the highest adversarial text fit score as the reference text adversarial suffix. Specifically, this embodiment employs a greedy search and gradient-guided discrete optimization algorithm for adversarial text suffixes. Since text has a discrete structure, gradient updates cannot be performed directly. This application adopts an optimization strategy combining greedy search and gradient guidance, with the specific steps as follows: i. Gradient localization: Adversarial suffix targeting of the current text Each word element Its sensitivity to text loss is calculated using the following seventh formula:

[0134] in, This represents the i-th word element in the current text's adversarial suffix. The text loss for the i-th term Sensitivity Let be the gradient of the text loss with respect to the i-th word; Select the location with maximum sensitivity. This indicates that the lexical element at that position has the greatest impact on the offensiveness, that is, the... The lexical position of the target lexical to be modified.

[0135] ii. Candidate generation: from a pre-defined vocabulary Select several target candidate modified words These target candidate modification terms need to be evaluated for effectiveness using two dimensions: semantic coherence value and attack effectiveness value. Specifically, the eighth formula can be used to determine the target candidate modification terms.

[0136] The eighth calculation formula satisfies the following formula:

[0137] in, Modify the word units for the target candidate. Q This is the original malicious command. T To pre-determine the attack target, For attack images, For text adversarial suffixes S Center front A string consisting of 10 characters As a weighting factor, For text concatenation, Modify the lexical set for all target candidates. topK To select the largest value K indivual, Indicates that in a given Under the given conditions, the probability of each word in the preset vocabulary being the next output character is... Represents the given text and images Under the condition of output The probability of S The Middle i each word element The gradient; That is, based on the eighth calculation formula, the weighted sum of the semantic coherence value and attack effectiveness value of each word in the preset vocabulary is determined, and the top K words with the largest weighted sum value are selected as target candidate modified words.

[0138] iii. Candidate Evaluation: Generating new suffix versions for each candidate. The weighted score of the aggression score and the language naturalness score is calculated using the following ninth formula. It should be noted that the weighted score of the aggression score and the language naturalness score is also the adversarial text fit score.

[0139]

[0140] in, The adversarial text fit score is given by the adversarial suffix of the candidate text, where Q represents the original malicious instruction. S represents the candidate text adversarial suffix. As a weighting factor, For the given text Given an attack image I, output the probability of a preset attack target T. This term is used to constrain the aggressiveness. For a given Output target candidate modified words under the condition The probability is used to constrain the naturalness of the text. for The string preceding the position of the replaced word in the string.

[0141] iv. Update selection: Finally, select the candidate text version with the highest score as the optimization result of the current round and proceed to the next iteration.

[0142] In this embodiment, the key words that have the greatest impact on text loss are first located, then candidate words that simultaneously satisfy semantic coherence and attack effectiveness are selected for replacement, and finally the optimal suffix is ​​selected through fit score.

[0143] In some implementations, when generating a model response to the original malicious instruction based on the multimodal embedding fusion features and a jailbreak attack evaluation type for the model response, a preset multimodal large language model can be used to perform multimodal semantic collaborative processing on the multimodal embedding fusion features to obtain a language generation result for the original malicious instruction. The language generation result is then used as the model response. It is then determined whether the model response meets the jailbreak attack success conditions. If the model response meets the jailbreak attack success conditions, the jailbreak attack evaluation type of the model response is a successful jailbreak attack. If the model response does not meet the jailbreak attack success conditions, the jailbreak attack evaluation type of the model response is a failed jailbreak attack. The jailbreak attack success conditions are that the model response includes a preset attack target, the semantic similarity between the model response and the original malicious instruction is greater than or equal to a preset semantic similarity threshold, and the perplexity of the model response is less than or equal to a preset perplexity threshold.

[0144] Among them, multimodal semantic collaborative processing refers to the process by which the preset multimodal large language model comprehensively analyzes the embedded features of different modalities such as text and vision, mines and integrates cross-modal semantic associations, and finally generates natural language results that simultaneously reflect the core information of multiple modalities.

[0145] Among them, the language generation result refers to the natural language text generated by the preset multimodal large language model based on multimodal embedding fusion features. It is the direct source of the model response and must simultaneously meet the dual requirements of conveying the original malicious intent and having concealment to circumvent review.

[0146] Among them, the successful jailbreak attack conditions refer to the core criteria for determining whether the model response is effective. It consists of three sub-conditions: "containing a preset attack target", "semantic proximity meets the standard", and "confusion meets the standard". It ensures the practicality of the attack text from three dimensions: completeness of intent, semantic consistency, and naturalness of language.

[0147] Semantic proximity refers to the quantification of the strength of the association between the model response and the original malicious instruction in terms of core semantics (especially malicious intent). The value range is usually 0-1, and the higher the value, the stronger the association.

[0148] The preset semantic proximity threshold refers to the minimum standard of semantic proximity set in advance, which is used to filter attack texts that have not deviated from the core intent of the original malicious instructions. Texts below this threshold are excluded because they cannot effectively convey malicious information.

[0149] Among them, perplexity is an indicator that measures the naturalness of the language in a text. It is calculated based on the prediction probability of each word in the text by a pre-trained language model. The lower the value, the more the text conforms to the normal human expression logic (such as grammatical fluency and logical coherence).

[0150] The preset perplexity threshold refers to the highest standard of perplexity set in advance, which is used to filter attack texts that meet the standard of language naturalness. Texts that exceed this threshold may have grammatical errors, logical confusion and other problems, and are easily identified as abnormal content by the review system.

[0151] In this embodiment, multimodal embedding and fusion features are input into a preset multimodal large language model. This preset multimodal large language model performs multimodal semantic collaborative processing, that is, deeply associating textual and visual semantics, integrating cross-modal association information into coherent natural language, generating a language generation result, and using it as the model response. After generating the model response, its validity needs to be determined by the conditions for a successful jailbreak attack; three sub-conditions must be met simultaneously: Condition 1: The model response includes a preset attack target: The preset attack target is the core malicious intent extracted from the original malicious instructions. It is necessary to verify whether the attack text contains the preset attack target or conveys the target through covert expression.

[0152] Condition 2: The semantic similarity between the model response and the original malicious instruction is greater than or equal to the preset semantic similarity threshold: If the semantic similarity between the model response and the original instruction reaches or exceeds the preset semantic similarity threshold, it means that the model response has not deviated from the core malicious intent; otherwise, it is judged as invalid due to the ambiguity of intent transmission.

[0153] Condition 3: The perplexity of the model response is less than or equal to the preset perplexity threshold: If the perplexity of the attack text is lower than this threshold, it means that the text reads naturally and smoothly and is not easily identified as malicious splicing content by the review system; otherwise, it is easily marked because the sentences are stiff and the logic is abrupt.

[0154] If all three conditions are met, the attack text is evaluated as a successful jailbreak attack; if any one condition is not met, the jailbreak attack fails.

[0155] Next, based on the multimodal large-model jailbreak risk detection and defense method described above, to make it easier for those skilled in the art to understand and implement the technical solution of this application, an example is given below to illustrate the generation of a jailbreak attack scheme, combined with a practical application scenario: Please refer to... Figure 3 , Figure 3 This is a schematic diagram illustrating the process of generating a jailbreak attack scheme provided in an embodiment of this application. Figure 3 The flowchart provided includes the following key steps: 1. Semantic parsing and initial embedding generation of modality: First, the original malicious command is semantically parsed to extract its key attack targets and semantic structure, and based on this, a set of image generation prompt words suitable for the text-based graph model is constructed. Then, the text encoder inside the text-based graph model is used to convert the prompt words into prompt embedding features, and at the same time, a latent vector for image generation is randomly initialized using a standard normal initialization strategy as the initial input state for the image generation process.

[0156] 2. Cross-modal adversarial example generation: In the text path, the original malicious command is appended with an initial adversarial suffix to form a complete attack text, which is then fed into the text encoder of a preset multimodal large language model to generate text embedding features for subsequent fusion. In the image path, the cue embedding features obtained in step 1 and the image generation latent vector are input into the text-to-image model to generate an attack image. Subsequently, the attack image is input into the visual encoder of the preset multimodal large language model to extract the corresponding visual embedding features.

[0157] 3. Multimodal fusion and response prediction: The text embedding features and visual embedding features obtained in step 2 are fused (e.g., concatenated or weighted) and then input into the language model layer of a pre-defined multimodal large language model to trigger the natural language understanding and generation process. The language model then generates a text output, i.e., the model response, used to evaluate whether the jailbreak attack was successful.

[0158] This section explicitly feeds the image modality and text modality into the visual encoder and text encoder of a pre-defined multimodal large language model for processing, respectively. Then, both are embedded into the language model layer of the vector fusion input model to drive the generation of attack output. Through this clearly defined modal embedding structure design, the attack intent is conveyed through dual channels in both semantic and visual spaces, significantly improving the efficiency of image-text linkage generation and the accuracy of the attack.

[0159] 4. Triple loss calculation: To improve the effectiveness of text-based collaborative jailbreaks, this application designs three types of loss functions: a) Attack effectiveness loss function: The cross-entropy between the language-generated result and the preset attack target is used to constrain the attack effectiveness; b) Semantic similarity loss function: The cosine similarity between the generated image and the original malicious instruction is calculated using the CLIP model to constrain the semantic fidelity of the image; c) Semantic coherence loss function: calculates the perplexity of adversarial suffixes, and measures the linguistic naturalness and semantic fluency of the entire text input after adding adversarial suffixes.

[0160] This section designs an optimization objective comprised of attack effectiveness loss, image-text semantic similarity loss, and semantic coherence loss. This objective simultaneously constrains the attack effectiveness and semantic preservation of the image modality, as well as the linguistic naturalness and stealth of the text modality, ensuring a balance between attack success rate and sample usability during the optimization process. For details on the three types of loss functions mentioned above, please refer to the corresponding sections above; they will not be repeated here.

[0161] 5. Alternating gradient updates: After calculating the loss function, this application employs a modality separation and alternating optimization strategy to update the two types of attack paths step by step: a) Latent Update: latent represents the latent representation of image generation, i.e., the latent vector of image generation, which is a differentiable parameter in a continuous vector space. This application adopts an optimization method based on gradient descent, using an image optimization objective composed of attack loss and semantic similarity loss, to backpropagate and update the latent vector, guiding the image to improve both attack power and semantic consistency; b) Adversarial Suffix Update: Adversarial suffixes consist of discrete lexical units, making them difficult to optimize directly using standard continuous gradient descent. Therefore, this application employs a strategy combining greedy search and gradient-based discrete optimization. It utilizes a text optimization objective comprised of attack loss and semantic coherence loss to backpropagate and update adversarial suffixes, guiding the text modality to maintain linguistic fluency while concealing attack intent.

[0162] Considering the different modal input characteristics, this application designs a separate optimization path for image-generated latent vectors and text adversarial suffixes: For image-generated latent vectors, a projective gradient descent attack strategy is employed, updating the vectors in a step-by-step manner based on the gradient direction within a continuous latent space; for text adversarial suffixes, a combination of greedy search and gradient guidance is used for discrete optimization based on token replacement, balancing attack strength and linguistic naturalness. This differentiated optimization strategy significantly improves attack stability and semantic controllability, and is a key technical means for enhancing attack quality in this application.

[0163] 6. Multi-round iterative optimization: Repeat steps 2 through 5, iterating through multiple rounds to progressively optimize the latent vectors generated from the image and the adversarial suffixes in the text, until the convergence thresholds for attack effectiveness, semantic matching, and language naturalness are met. The final output image and text, as joint input, can be used to induce the target multimodal large language model to generate malicious responses under secure mechanisms, achieving the goal of a multimodal jailbreak attack. It should be noted that reaching the convergence threshold means that the jailbreak attack evaluation type of the model response is a successful jailbreak attack.

[0164] The proposed method for detecting and defending against jailbreak risks in multimodal large language models based on image-text co-optimization effectively improves the success rate, concealment, and model generalization ability of jailbreak attacks by jointly optimizing adversarial suffixes in the text modality and latent variables in the image modality. It overcomes the technical bottlenecks of existing methods, which suffer from unstable attack effects and difficulty in breaking through multimodal defense mechanisms under single-modal attacks or modal decoupling. This method has significant technical advantages and application prospects.

[0165] First, in a white-box attack scenario, experiments show that this application can significantly improve the success rate of jailbreak attacks on multimodal models. As shown in Table 1, using four image-to-text models—Qwen2-VL-7B, llava-1.5-7b, llava-v1.6-mistral-7b, and MiniGPT-4—as attack targets, the success rate of attacks using adversarial images and adversarial commands obtained through joint optimization reached 94% to 100%. In comparison, the success rate of image-based single-modal attacks using original commands + adversarial images was only 14-21%, proving that simple image perturbation is difficult to break through security mechanisms; the success rate of text-based single-modal attacks using adversarial commands + original images was 68-83%, which is better than image attacks but still has a security gap of about 20%; while joint optimization brought the attack success rate close to 100%, verifying the decisive role of the cross-modal collaborative mechanism of text covert guidance + precise image delivery in attack effectiveness.

[0166]

[0167] Table 1. Comparison of white-box attack success rates (%) Secondly, this application also has a significant advantage in terms of concealment. The naturalness of the adversarial command's language was evaluated using a language model perplexity index. The results show that the perplexity of the adversarial command is 123.9±115.4, slightly higher than the original command's 80.8±73.9, but still within the acceptable range of natural language. It is far below the perplexity threshold of traditional text attacks (usually >1000), and does not introduce grammatical or semantic anomalies, demonstrating strong language concealment. Simultaneously, the semantic similarity between the adversarial image and the original malicious command, calculated using the CLIP model, reaches 0.32±0.07, indicating that the generated image maintains a reasonable visual semantic relevance to the original command's theme, effectively preventing attack failure due to semantic deviation of the image modality.

[0168] Furthermore, as shown in Table 2, when adversarial examples generated based on a single model are transferred to other models, this application still maintains an attack success rate of 43-87%. Adversarial graph inputs generated under the condition that the source model is Qwen2-VL-7B can be successfully transferred to target models with different structures and training mechanisms, such as llava-1.5-7b, llava-v1.6-mistral-7b, and MiniGPT-4, for attacks. Specifically, the transfer attack success rate on llava-1.5-7b reaches 87%, while it reaches 71% on llava-v1.6-mistral-7b and 43% on MiniGPT-4, demonstrating that the adversarial examples generated in this application possess strong cross-model attack capabilities and generalization robustness, which is beneficial for general jailbreak testing and security assessment in real-world environments.

[0169]

[0170] Table 2 Success Rate of Cross-Model Migration Attacks (%) In summary, this application achieves significant improvements in attack effectiveness, input concealment, and model transferability by introducing a modal joint optimization strategy. It provides a novel technical path and practical basis for the security assessment and adversarial sample generation of multimodal large language models, and has high theoretical value and engineering application potential.

[0171] One embodiment also provides a multimodal large-scale jailbreak risk detection and defense device. Please refer to [link / reference]. Figure 4 , Figure 4 This is a schematic diagram of the structure of the multimodal large-scale jailbreak risk detection and defense device 200 provided in this application embodiment. The multimodal large-scale jailbreak risk detection and defense device 200 is applied to electronic devices and includes an acquisition module 201, a construction module 202, a generation module 203, a determination module 204, and an anti-jailbreak attack module 205, as follows: Module 201 is used to acquire the original malicious instructions; Construction module 202 is used to construct visual embedding features based on the original malicious instructions, and to construct text embedding features based on the original malicious instructions; The generation module 203 is used to determine multimodal embedding fusion features based on the text embedding features and the visual embedding features, and to generate a model response to the original malicious instruction and a jailbreak attack evaluation type of the model response based on the multimodal embedding fusion features. The determination module 204 is used to determine the jailbreak attack scheme against the original malicious command if the jailbreak attack evaluation type of the model response is a successful jailbreak attack.

[0172] The anti-jailbreak attack module 205 is used to perform model jailbreak risk detection and defense processing based on the jailbreak attack scheme.

[0173] In some implementations, the construction module 202 is used for: The original malicious instruction is semantically parsed to construct image generation prompt words, the image generation prompt words are converted into prompt embedding features, and the image generation latent vector is randomly initialized using a standard normal initialization strategy. An attack image is generated based on the aforementioned prompt embedding features and the aforementioned image generation latent vectors. Visual feature extraction is then performed on the attack image to obtain visual embedding features targeting the original malicious instruction. Obtain the text adversarial suffix for the original malicious instruction, append the text adversarial suffix to the original malicious instruction to obtain the attack text, extract text features from the attack text to obtain the text embedding features for the original malicious instruction.

[0174] In some embodiments, the construction module 202 is further configured to: If the jailbreak attack evaluation type of the model response is jailbreak attack failure, then update the visual embedding features and / or update the text embedding features, and perform the step of determining multimodal embedding fusion features based on the text embedding features and the visual embedding features.

[0175] In some implementations, the construction module 202 is used for: The steps include updating the image generation latent vector, performing image generation processing based on the prompt embedding features and the image generation latent vector to obtain an attack image, and performing visual feature extraction processing based on the attack image to obtain visual embedding features for the original malicious instruction.

[0176] In some implementations, the construction module 202 is used for: The preset attack target is obtained through the original malicious command; The attack effectiveness loss is obtained using a first calculation formula based on the model response and the preset attack target. The semantic similarity loss is obtained using a second calculation formula based on the original malicious instructions and attack images; The latent variable loss is obtained by using a third calculation formula based on the attack effectiveness loss and the semantic similarity loss, and the latent vector is generated from the reference image based on the latent variable loss. The latent vector of the current image is updated based on the latent vector generated from the reference image; The first calculation formula satisfies the following formula:

[0177] in, R represents the attack effectiveness loss, R represents the model response, and T represents the preset attack target. The cross-entropy loss function; The second calculation formula satisfies the following formula:

[0178] in, The semantic similarity loss is used, where Q represents the original malicious command and I represents the attack image. For text feature extraction functions, This is an image feature extraction function; The third calculation formula satisfies the following formula:

[0179] in, For latent variable loss, For adjustable hyperparameters, .

[0180] In some implementations, the construction module 202 is used for: Based on the latent variable loss, the sixth calculation formula is used to update the vector and obtain the updated reference image to generate the latent vector; The sixth calculation formula satisfies the following formula:

[0181] in, For the first The reference image of the wheel generates latent vectors. For the first The updated reference image of the wheel generates latent vectors, where t is a positive integer. Step size, for Constraint domain This indicates a projection operation. For the latent variable loss on the first The gradient of the latent vector is generated from the reference image.

[0182] In some implementations, the construction module 202 is used for: The steps are as follows: update the adversarial suffix of the text, append the adversarial suffix to the original malicious instruction to obtain the attack text, extract text features from the attack text, and obtain the text embedding features for the original malicious instruction.

[0183] In some implementations, the construction module 202 is used for: The preset attack target is obtained through the original malicious command; The attack effectiveness loss is obtained using a first calculation formula based on the model response and the preset attack target. The semantic coherence loss is obtained by using a fourth computational formula based on text adversarial suffixes; The text loss is obtained using the fifth calculation formula based on the attack effectiveness loss and the semantic coherence loss, and the reference text adversarial suffix is ​​obtained based on the text loss. The current text adversarial suffix is ​​updated based on the reference text adversarial suffix. The first calculation formula satisfies the following formula:

[0184] in, R represents the attack effectiveness loss, R represents the model response, and T represents the preset attack target. The cross-entropy loss function; The fourth calculation formula satisfies the following formula:

[0185] in, The loss is for semantic coherence, where n is the length of the sequence. For conditional probability, It is an exponential function. This represents the i-th word in the sequence, where i is a positive integer; The fifth calculation formula satisfies the following formula:

[0186] in, For text loss, For adjustable hyperparameters, .

[0187] In some implementations, the construction module 202 is used for: The sensitivity of each lexical unit in the current text adversarial suffix to the text loss is calculated based on the seventh calculation formula, and the target lexical unit to be modified corresponding to the position of maximum sensitivity is determined. Based on the eighth calculation formula, the target candidate modified word is determined by the semantic coherence value and attack effectiveness value of each candidate modified word in the preset vocabulary; Candidate text adversarial suffixes are generated by replacing the target word to be modified in the current text adversarial suffix with the target candidate modified word; Based on the ninth calculation formula, the adversarial text fit score of the candidate text adversarial suffix is ​​determined, and the candidate text adversarial suffix corresponding to the highest adversarial text fit score is used as the reference text adversarial suffix. The seventh calculation formula satisfies the following formula:

[0188] in, This represents the i-th word element in the current text's adversarial suffix. The text loss for the i-th term Sensitivity Let be the gradient of the text loss with respect to the i-th word; The eighth calculation formula satisfies the following formula:

[0189] in, Modify the word units for the target candidate. Q This is the original malicious command. T To pre-determine the attack target, For attack images, For text adversarial suffixes S Center front A string consisting of 10 characters As a weighting factor, For text concatenation, Modify the lexical set for all target candidates. topK To select the largest value K indivual, Indicates that in a given Under the given conditions, the probability of each word in the preset vocabulary being the next output character is... Represents the given text and images Under the condition of output The probability of S The Middle i each word element The gradient; The ninth calculation formula satisfies the following formula:

[0190] Where I represents the attack image, T represents the preset attack target, and Q represents the original malicious command. As candidate text adversarial suffixes, S represents the adversarial text fit score of the candidate text adversarial suffix, where S is the text adversarial suffix. As a weighting factor, For text concatenation, For the given text Given the attack image, output the probability of the preset attack target. Modify the word units for the target candidate. The lexical position of the target lexical to be modified. , For a given Under certain conditions, the probability of outputting the target candidate modified words is used to constrain the naturalness of the text. for The string preceding the position corresponding to the replaced word in the text.

[0191] In some embodiments, the generation module 203 is used for: By performing multimodal semantic collaborative processing on the multimodal embedding and fusion features through a preset multimodal large language model, a language generation result for the original malicious instruction is obtained, and the language generation result is used as the model response. Determine whether the model response meets the conditions for a successful jailbreak attack; If the model response meets the jailbreak attack success condition, then the jailbreak attack evaluation type of the model response is jailbreak attack success; If the model response does not meet the conditions for a successful jailbreak attack, then the jailbreak attack evaluation type of the model response is jailbreak attack failure. The conditions for a successful jailbreak attack are that the model response includes a preset attack target, the semantic similarity between the model response and the original malicious instruction is greater than or equal to a preset semantic similarity threshold, and the perplexity of the model response is less than or equal to a preset perplexity threshold.

[0192] It should be noted that the multimodal large-scale jailbreak risk detection and defense device provided in this application embodiment belongs to the same concept as the multimodal large-scale jailbreak risk detection and defense method in the above embodiment. The multimodal large-scale jailbreak risk detection and defense device can implement any of the methods provided in the embodiment of the multimodal large-scale jailbreak risk detection and defense method. For details of its implementation process, please refer to the embodiment of the multimodal large-scale jailbreak risk detection and defense method, which will not be repeated here.

[0193] Furthermore, to better implement the multimodal large-model jailbreak risk detection and defense method in the embodiments of this application, based on the multimodal large-model jailbreak risk detection and defense method, this application also provides an electronic device. The electronic device can be any device equipped with a processor and possessing processing capabilities, such as mobile electronic devices with processors like smartphones, tablets, handheld computers, and laptops, or fixed electronic devices with processors like desktop computers, televisions, and servers. Please refer to [link / reference]. Figure 5 , Figure 5 This is a schematic diagram of a first structure of an electronic device provided in an embodiment of this application. The electronic device 300 includes a processor 301 and a memory 302. The processor 301 and the memory 302 are electrically connected.

[0194] The processor 301 is the control center of the electronic device 300. It connects various parts of the electronic device via various interfaces and lines, and executes various functions and processes data by running or calling computer programs stored in the memory 302 and accessing data stored in the memory 302, thereby providing overall monitoring of the electronic device. The processor 301 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0195] The memory 302 can be used to store computer programs and data. The computer programs stored in the memory 302 contain instructions that can be executed in the processor. The computer programs can be composed of various functional modules. The processor 401 executes various functional applications and data processing by calling the computer programs stored in the memory 302. The memory 302 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device 300 (such as audio data, video data, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0196] In this embodiment, the processor 301 in the electronic device 300 loads the instructions corresponding to the processes of one or more computer programs into the memory 302 according to the following steps, and the processor 401 runs the computer programs stored in the memory 302 to realize various functions: Obtain the original malicious commands; Visual embedding features are constructed based on the original malicious instructions, and text embedding features are constructed based on the original malicious instructions; Based on the text embedding features and the visual embedding features, multimodal embedding fusion features are determined, and based on the multimodal embedding fusion features, a model response to the original malicious command and a jailbreak attack evaluation type of the model response are generated. If the jailbreak attack evaluation type of the model response is jailbreak attack success, then the attack text corresponding to the text embedding feature and the attack image corresponding to the visual embedding feature are used as jailbreak attack schemes against the original malicious command. Based on the aforementioned jailbreak attack scheme, a model jailbreak risk detection and defense process is implemented.

[0197] In some implementations, please refer to Figure 6 , Figure 6This is a second structural schematic diagram of the electronic device provided in an embodiment of this application. The electronic device 300 further includes: a radio frequency circuit 303, a display screen 304, a control circuit 305, an input unit 306, an audio circuit 307, a sensor 308, and a power supply 309. The processor 301 is electrically connected to the radio frequency circuit 303, the display screen 304, the control circuit 305, the input unit 306, the audio circuit 307, the sensor 308, and the power supply 309.

[0198] The radio frequency circuit 303 is used to transmit and receive radio frequency signals to communicate with network devices or other electronic devices via wireless communication.

[0199] The display screen 304 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of electronic devices, which can be composed of images, text, icons, videos, and any combination thereof.

[0200] The control circuit 305 is electrically connected to the display screen 304 and is used to control the display screen 304 to display information.

[0201] The input unit 306 can be used to receive input numeric or character information or user characteristic information (such as fingerprints), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control. The input unit 306 may include a fingerprint recognition module.

[0202] The audio circuit 307 provides an audio interface between the user and the electronic device via a speaker and a microphone. The audio circuit 307 includes a microphone, which is electrically connected to the processor 301. The microphone is used to receive voice information input by the user.

[0203] Sensor 308 is used to collect information about the external environment. Sensor 308 may include one or more sensors such as an ambient light sensor, an accelerometer, and a gyroscope.

[0204] The power supply 309 is used to supply power to the various components of the electronic device 300. In some embodiments, the power supply 309 can be logically connected to the processor 301 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system.

[0205] Although not shown in the figure, the electronic device 300 may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0206] In this embodiment, the processor 301 in the electronic device 300 loads the instructions corresponding to the processes of one or more computer programs into the memory 302 according to the following steps, and the processor 301 runs the computer programs stored in the memory 302 to realize various functions: Obtain the original malicious commands; Visual embedding features are constructed based on the original malicious instructions, and text embedding features are constructed based on the original malicious instructions; Based on the text embedding features and the visual embedding features, multimodal embedding fusion features are determined, and based on the multimodal embedding fusion features, a model response to the original malicious command and a jailbreak attack evaluation type of the model response are generated. If the jailbreak attack evaluation type of the model response is jailbreak attack success, then the attack text corresponding to the text embedding feature and the attack image corresponding to the visual embedding feature are used as jailbreak attack schemes against the original malicious command. Based on the aforementioned jailbreak attack scheme, a model jailbreak risk detection and defense process is implemented.

[0207] In some implementations, when processor 301 executes the construction of visual embedding features based on the original malicious instructions and the construction of text embedding features based on the original malicious instructions, it may perform the following: The original malicious instruction is semantically parsed to construct image generation prompt words, the image generation prompt words are converted into prompt embedding features, and the image generation latent vector is randomly initialized using a standard normal initialization strategy. An attack image is generated based on the aforementioned prompt embedding features and the aforementioned image generation latent vectors. Visual feature extraction is then performed on the attack image to obtain visual embedding features targeting the original malicious instruction. Obtain the text adversarial suffix for the original malicious instruction, append the text adversarial suffix to the original malicious instruction to obtain the attack text, extract text features from the attack text to obtain the text embedding features for the original malicious instruction.

[0208] In some implementations, processor 301 may also perform: If the jailbreak attack evaluation type of the model response is jailbreak attack failure, then update the visual embedding features and / or update the text embedding features, and perform the step of determining multimodal embedding fusion features based on the text embedding features and the visual embedding features.

[0209] In some implementations, when processor 301 performs the update of the visual embedding features, it may perform the following: The steps include updating the image generation latent vector, performing image generation processing based on the prompt embedding features and the image generation latent vector to obtain an attack image, and performing visual feature extraction processing based on the attack image to obtain visual embedding features for the original malicious instruction.

[0210] In some implementations, when processor 301 performs the update of the image generating latent vector, it may perform the following: The preset attack target is obtained through the original malicious command; The attack effectiveness loss is obtained using a first calculation formula based on the model response and the preset attack target. The semantic similarity loss is obtained using a second calculation formula based on the original malicious instructions and attack images; The latent variable loss is obtained by using a third calculation formula based on the attack effectiveness loss and the semantic similarity loss, and the latent vector is generated from the reference image based on the latent variable loss. The latent vector of the current image is updated based on the latent vector generated from the reference image; The first calculation formula satisfies the following formula:

[0211] in, R represents the attack effectiveness loss, R represents the model response, and T represents the preset attack target. The cross-entropy loss function; The second calculation formula satisfies the following formula:

[0212] in, The semantic similarity loss is used, where Q represents the original malicious command and I represents the attack image. For text feature extraction functions, This is an image feature extraction function; The third calculation formula satisfies the following formula:

[0213] in, For latent variable loss, For adjustable hyperparameters, .

[0214] In some implementations, when processor 301 executes the process of generating latent vectors from the reference image based on the latent variable loss, it may perform the following: Based on the latent variable loss, the sixth calculation formula is used to update the vector and obtain the updated reference image to generate the latent vector; The sixth calculation formula satisfies the following formula:

[0215] in, For the first The reference image of the wheel generates latent vectors. For the first The updated reference image of the wheel generates latent vectors, where t is a positive integer. Step size, for Constraint domain This indicates a projection operation. For the latent variable loss on the first The gradient of the latent vector is generated from the reference image.

[0216] In some implementations, when processor 301 performs the update of the text embedding features, it may perform the following: The steps are as follows: update the adversarial suffix of the text, append the adversarial suffix to the original malicious instruction to obtain the attack text, extract text features from the attack text, and obtain the text embedding features for the original malicious instruction.

[0217] In some implementations, when processor 301 performs the update of the text adversarial suffix, it may perform the following: The preset attack target is obtained through the original malicious command; The attack effectiveness loss is obtained using a first calculation formula based on the model response and the preset attack target. The semantic coherence loss is obtained by using a fourth computational formula based on text adversarial suffixes; The text loss is obtained using the fifth calculation formula based on the attack effectiveness loss and the semantic coherence loss, and the reference text adversarial suffix is ​​obtained based on the text loss. The current text adversarial suffix is ​​updated based on the reference text adversarial suffix. The first calculation formula satisfies the following formula:

[0218] in, R represents the attack effectiveness loss, R represents the model response, and T represents the preset attack target. The cross-entropy loss function; The fourth calculation formula satisfies the following formula:

[0219] in, The loss is for semantic coherence, where n is the length of the sequence. For conditional probability, It is an exponential function. This represents the i-th word in the sequence, where i is a positive integer; The fifth calculation formula satisfies the following formula:

[0220] in, For text loss, For adjustable hyperparameters, .

[0221] In some implementations, when processor 301 performs the process of obtaining the reference text adversarial suffix based on the text loss, it may perform the following: The sensitivity of each lexical unit in the current text adversarial suffix to the text loss is calculated based on the seventh calculation formula, and the target lexical unit to be modified corresponding to the position of maximum sensitivity is determined. Based on the eighth calculation formula, the target candidate modified word is determined by the semantic coherence value and attack effectiveness value of each candidate modified word in the preset vocabulary; Candidate text adversarial suffixes are generated by replacing the target word to be modified in the current text adversarial suffix with the target candidate modified word; Based on the ninth calculation formula, the adversarial text fit score of the candidate text adversarial suffix is ​​determined, and the candidate text adversarial suffix corresponding to the highest adversarial text fit score is used as the reference text adversarial suffix. The seventh calculation formula satisfies the following formula:

[0222] in, This represents the i-th word element in the current text's adversarial suffix. The text loss for the i-th term Sensitivity Let be the gradient of the text loss with respect to the i-th word; The eighth calculation formula satisfies the following formula:

[0223] in, Modify the word units for the target candidate. Q This is the original malicious command. T To pre-determine the attack target, For attack images, For text adversarial suffixes S Center front A string consisting of 10 characters As a weighting factor, For text concatenation, Modify the lexical set for all target candidates. topK To select the largest value K indivual, Indicates that in a given Under the given conditions, the probability of each word in the preset vocabulary being the next output character is... Represents the given text and images Under the condition of output The probability of S The Middle i each word element The gradient; The ninth calculation formula satisfies the following formula:

[0224] Where I represents the attack image, T represents the preset attack target, and Q represents the original malicious command. As candidate text adversarial suffixes, S represents the adversarial text fit score of the candidate text adversarial suffix, where S is the text adversarial suffix. As a weighting factor, For text concatenation, For the given text Given the attack image, output the probability of the preset attack target. Modify the word units for the target candidate. The lexical position of the target lexical to be modified. , For a given Under certain conditions, the probability of outputting the target candidate modified words is used to constrain the naturalness of the text. for The string preceding the position corresponding to the replaced word in the text.

[0225] In some implementations, when processor 301 executes the generation of a model response to the original malicious instruction based on the multimodal embedding fusion features and the jailbreak attack evaluation type of the model response, it may perform the following: By performing multimodal semantic collaborative processing on the multimodal embedding and fusion features through a preset multimodal large language model, a language generation result for the original malicious instruction is obtained, and the language generation result is used as the model response. Determine whether the model response meets the conditions for a successful jailbreak attack; If the model response meets the jailbreak attack success condition, then the jailbreak attack evaluation type of the model response is jailbreak attack success; If the model response does not meet the conditions for a successful jailbreak attack, then the jailbreak attack evaluation type of the model response is jailbreak attack failure. The conditions for a successful jailbreak attack are that the model response includes a preset attack target, the semantic similarity between the model response and the original malicious instruction is greater than or equal to a preset semantic similarity threshold, and the perplexity of the model response is less than or equal to a preset perplexity threshold.

[0226] This application also provides a computer-readable storage medium storing a computer program. When the computer program is run on a computer, the computer executes the multimodal large-scale jailbreak risk detection and defense method described in any of the above embodiments.

[0227] It should be noted that those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, which may include, but is not limited to, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0228] This application also provides a computer program product containing instructions that, when the computer program product is run on a computer or processor, cause the computer or processor to execute the multimodal large-model jailbreak risk detection and defense method described in any of the above embodiments.

[0229] Furthermore, the terms "first," "second," and "third," etc., used in this application are used to distinguish different objects, not to describe a specific order. Additionally, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not limited to the listed steps or modules, but some embodiments may also include steps or modules not listed, or some embodiments may include other steps or modules inherent to these processes, methods, products, or devices.

[0230] The multimodal large-model jailbreak risk detection and defense method and electronic device provided in the embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and its core ideas; at the same time, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for detecting and defending against multimodal large-scale jailbreak risks, characterized in that, include: Obtain the original malicious commands; Visual embedding features are constructed based on the original malicious instructions, and text embedding features are constructed based on the original malicious instructions; Based on the text embedding features and the visual embedding features, multimodal embedding fusion features are determined, and based on the multimodal embedding fusion features, a model response to the original malicious command and a jailbreak attack evaluation type of the model response are generated. If the jailbreak attack evaluation type of the model response is jailbreak attack success, then the attack text corresponding to the text embedding feature and the attack image corresponding to the visual embedding feature are used as jailbreak attack schemes against the original malicious command. Based on the aforementioned jailbreak attack scheme, a model jailbreak risk detection and defense process is implemented.

2. The method according to claim 1, characterized in that, The construction of visual embedding features based on the original malicious instructions, and the construction of text embedding features based on the original malicious instructions, include: The original malicious instruction is semantically parsed to construct image generation prompt words, the image generation prompt words are converted into prompt embedding features, and the image generation latent vector is randomly initialized using a standard normal initialization strategy. An attack image is generated based on the aforementioned prompt embedding features and the aforementioned image generation latent vectors. Visual feature extraction is then performed on the attack image to obtain visual embedding features targeting the original malicious instruction. Obtain the text adversarial suffix for the original malicious instruction, append the text adversarial suffix to the original malicious instruction to obtain the attack text, extract text features from the attack text to obtain the text embedding features for the original malicious instruction.

3. The method according to claim 2, characterized in that, The method further includes: If the jailbreak attack evaluation type of the model response is jailbreak attack failure, then update the visual embedding features and / or update the text embedding features, and perform the step of determining multimodal embedding fusion features based on the text embedding features and the visual embedding features.

4. The method according to claim 3, characterized in that, The updating of the visual embedding features includes: The steps include updating the image generation latent vector, performing image generation processing based on the prompt embedding features and the image generation latent vector to obtain an attack image, and performing visual feature extraction processing based on the attack image to obtain visual embedding features for the original malicious instruction.

5. The method according to claim 4, characterized in that, The updating of the latent vector generated from the image includes: The preset attack target is obtained through the original malicious command; The attack effectiveness loss is obtained using a first calculation formula based on the model response and the preset attack target. The semantic similarity loss is obtained using a second calculation formula based on the original malicious instructions and attack images; The latent variable loss is obtained by using a third calculation formula based on the attack effectiveness loss and the semantic similarity loss, and the latent vector is generated from the reference image based on the latent variable loss. The latent vector of the current image is updated based on the latent vector generated from the reference image; The first calculation formula satisfies the following formula: in, R represents the attack effectiveness loss, R represents the model response, and T represents the preset attack target. The cross-entropy loss function; The second calculation formula satisfies the following formula: in, The semantic similarity loss is used, where Q represents the original malicious command and I represents the attack image. For text feature extraction functions, This is an image feature extraction function; The third calculation formula satisfies the following formula: in, For latent variable loss, For adjustable hyperparameters, .

6. The method according to claim 5, characterized in that, The step of generating latent vectors from the reference image based on the latent variable loss includes: Based on the latent variable loss, the sixth calculation formula is used to update the vector and obtain the updated reference image to generate the latent vector; The sixth calculation formula satisfies the following formula: in, For the first The reference image of the wheel generates latent vectors. For the first The updated reference image of the wheel generates latent vectors, where t is a positive integer. Step size, for Constraint domain This indicates a projection operation. For the latent variable loss on the first The gradient of the latent vector is generated from the reference image.

7. The method according to claim 3, characterized in that, Updating the text embedding features includes: The steps are as follows: update the adversarial suffix of the text, append the adversarial suffix to the original malicious instruction to obtain the attack text, extract text features from the attack text, and obtain the text embedding features for the original malicious instruction.

8. The method according to claim 7, characterized in that, The updating of the text adversarial suffix includes: The preset attack target is obtained through the original malicious command; The attack effectiveness loss is obtained using a first calculation formula based on the model response and the preset attack target. The semantic coherence loss is obtained by using a fourth computational formula based on text adversarial suffixes; The text loss is obtained using the fifth calculation formula based on the attack effectiveness loss and the semantic coherence loss, and the reference text adversarial suffix is ​​obtained based on the text loss. The current text adversarial suffix is ​​updated based on the reference text adversarial suffix. The first calculation formula satisfies the following formula: in, R represents the attack effectiveness loss, R represents the model response, and T represents the preset attack target. The cross-entropy loss function; The fourth calculation formula satisfies the following formula: in, The loss is for semantic coherence, where n is the length of the sequence. For conditional probability, It is an exponential function. This represents the i-th word in the sequence, where i is a positive integer; The fifth calculation formula satisfies the following formula: in, For text loss, For adjustable hyperparameters, .

9. The method according to claim 8, characterized in that, The process of obtaining the reference text adversarial suffix based on the text loss includes: The sensitivity of each lexical unit in the current text adversarial suffix to the text loss is calculated based on the seventh calculation formula, and the target lexical unit to be modified corresponding to the position of maximum sensitivity is determined. Based on the eighth calculation formula, the target candidate modified word is determined by the semantic coherence value and attack effectiveness value of each candidate modified word in the preset vocabulary; Candidate text adversarial suffixes are generated by replacing the target word to be modified in the current text adversarial suffix with the target candidate modified word; Based on the ninth calculation formula, the adversarial text fit score of the candidate text adversarial suffix is ​​determined, and the candidate text adversarial suffix corresponding to the highest adversarial text fit score is used as the reference text adversarial suffix. The seventh calculation formula satisfies the following formula: in, This represents the i-th word element in the current text's adversarial suffix. The text loss for the i-th term Sensitivity Let be the gradient of the text loss with respect to the i-th word; The eighth calculation formula satisfies the following formula: in, Modify the word units for the target candidate. Q This is the original malicious command. T To pre-determine the attack target, For attack images, For text adversarial suffixes S Center front A string consisting of 10 characters As a weighting factor, For text concatenation, Modify the lexical set for all target candidates. topK To select the largest value K indivual, Indicates that in a given Under the given conditions, the probability of each word in the preset vocabulary being the next output character is... Represents the given text and images Under the condition of output The probability of S The Middle i each word element The gradient; The ninth calculation formula satisfies the following formula: Where I represents the attack image, T represents the preset attack target, and Q represents the original malicious command. As candidate text adversarial suffixes, S represents the adversarial text fit score of the candidate text adversarial suffix, where S is the text adversarial suffix. As a weighting factor, For text concatenation, For the given text Given the attack image, output the probability of the preset attack target. Modify the word units for the target candidate. The lexical position of the target lexical to be modified. , For a given Under certain conditions, the probability of outputting the target candidate modified words is used to constrain the naturalness of the text. for The string preceding the position corresponding to the replaced word in the text.

10. An electronic device comprising a processor and a memory, the memory storing a computer program, characterized in that, The processor invokes the computer program to execute the multimodal large-model jailbreak risk detection and defense method as described in any one of claims 1 to 9.

Images