Model security evaluation method, device and storage medium

Abstract

The application discloses a model security evaluation method and device and a storage medium. The method comprises the following steps: obtaining a prompt word containing model attack content for generating an agent output, and obtaining an initial prompt word; inputting the initial prompt word into a defense agent for attack analysis, and obtaining rebuttal opinions corresponding to the initial prompt word; determining the model attack content recognized in the prompt word based on the rebuttal opinions, optimizing the model attack content recognized in the prompt word, and obtaining a model attack prompt word; and sending the model attack prompt word to a model to be evaluated for security evaluation. The generation agent automatically adjusts the concealment and logical rigor of the attack strategy according to the real-time feedback of the defense agent, directly repairs the attack vulnerabilities in the prompt word, generates a model attack prompt word with anti-vulnerability, improves the quality of the model attack prompt word, and can automatically, continuously and efficiently find the security vulnerabilities of the model to be evaluated.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a model security assessment method, device and storage medium. Background Technology

[0002] With the rapid development of large-scale model technology, its widespread application in fields such as natural language processing, image generation, and intelligent decision-making has brought significant efficiency improvements and innovation opportunities. However, the openness and generalization capabilities of large-scale models also present them with numerous security challenges. For example, models may generate false information, leak training data privacy, or be exploited by malicious attackers for jailbreaking attacks, information injection, and other malicious purposes. Furthermore, in practical deployments, large-scale models may be used to create phishing content, spread illegal information, or even be used for social engineering attacks, thereby posing potential harm to individuals, businesses, and society as a whole.

[0003] Therefore, how to automatically, continuously, and efficiently discover security vulnerabilities in large models has become a key issue in the development of artificial intelligence (AI). Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides at least one model security assessment method, device, and storage medium.

[0005] The first aspect of this application provides a model security assessment method. The prompt generator includes a generating agent and a defending agent. The method includes: obtaining prompt words containing model attack content output by the generating agent to obtain initial prompt words; inputting the initial prompt words into the defending agent for attack analysis to obtain rebuttal opinions corresponding to the initial prompt words; determining the model attack content identified in the initial prompt words based on the rebuttal opinions; optimizing the identified model attack content in the initial prompt words by the generating agent to obtain model attack prompt words; and sending the model attack prompt words to the model to be assessed to perform a security assessment on the model to be assessed based on the response results output by the model to be assessed.

[0006] In one embodiment, the generating agent includes a logical agent, an emotion masquerading agent, and an attack conversion agent; obtaining the prompt words containing model attack content output by the generating agent to obtain the initial prompt words includes: the logical agent generating an attack logic chain based on the target vulnerability; the emotion masquerading agent adding emotional semantics to the attack logic chain to obtain the emotion masquerading prompt; and the attack conversion agent converting the sensitive content in the emotion masquerading prompt into ambiguous semantics to obtain the initial prompt words.

[0007] In one embodiment, the generating agent optimizes the identified model attack content in the initial prompt words to obtain model attack prompt words, including: if the identified model attack content includes an attack logic chain, the logic agent optimizes the identified explicit features in the attack logic chain; if the identified model attack content includes sentiment semantics, the sentiment masquerading agent optimizes the identified explicit features in the sentiment semantics; if the identified model attack content includes attack sensitive words, the attack conversion agent optimizes the identified attack sensitive words.

[0008] In one embodiment, a base model is pre-trained. The logical agent, the emotion masquerading agent, the attack conversion agent, and the defense agent share the weights of the base model and are each loaded with independent low-rank adaptation weights, which are obtained by fine-tuning the weights of the base model.

[0009] In one embodiment, the generating agent optimizes the identified model attack content in the initial prompt to obtain a model attack prompt, including: the generating agent optimizes the identified model attack content in the initial prompt to obtain an optimized prompt; determines whether a preset termination condition is met; if not, the optimized prompt is input as a new initial prompt to the defense agent for attack analysis to obtain new rebuttal opinions; the generating agent continues to optimize the identified model attack content in the new initial prompt based on the new rebuttal opinions; if satisfied, the optimized prompt is used as a model attack prompt.

[0010] In one embodiment, the method further includes: inputting model attack prompts into multiple large language models to obtain the response results output by the multiple large language models respectively; and training a prompt generator based on the response results output by the multiple large language models respectively.

[0011] In one embodiment, a voting device is also included; the model attack prompts are input into multiple large language models to obtain the response results output by the multiple large language models, including: inputting each model attack prompt into the voting device for scoring to obtain an attack score corresponding to each model attack prompt; wherein the attack score is positively correlated with the danger level of the model attack prompt; selecting a preset number of model attack prompts with the highest attack scores, and inputting the selected model attack prompts into the multiple large language models to obtain the response results output by the multiple large language models.

[0012] In one embodiment, training a prompt generator based on the answer results output by multiple large language models includes: obtaining a dangerous semantic reference library, which includes multiple dangerous semantic statements; calculating the maximum semantic similarity between the answer result output by each large language model and the multiple dangerous semantic statements; and updating the prompt generator based on the training loss.

[0013] The second aspect of this application provides a model security assessment device, comprising: a prompt word acquisition module for acquiring prompt words containing model attack content output by a generating agent to obtain initial prompt words; a rebuttal module for inputting the initial prompt words into a defensive agent for attack analysis to obtain rebuttal opinions corresponding to the initial prompt words; an optimization module for determining the identified model attack content in the initial prompt words based on the rebuttal opinions, and optimizing the identified model attack content in the initial prompt words by the generating agent to obtain model attack prompt words; and an assessment module for sending the model attack prompt words to the model to be assessed to perform a security assessment on the model to be assessed based on the response results output by the model to be assessed.

[0014] A third aspect of this application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-described model security assessment method.

[0015] The fourth aspect of this application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implement the above-described model security assessment method.

[0016] The above scheme obtains initial prompts by acquiring prompts containing model attack content output by the generating agent. These initial prompts are then input into the defending agent for attack analysis, yielding corresponding rebuttal opinions. Based on these rebuttal opinions, the identified model attack content within the prompts is determined, and the identified model attack prompts are optimized to obtain model attack prompts. This achieves a dynamic cognitive game between the generating and defending agents. The generating agent automatically adjusts the stealth and logical rigor of its attack strategy based on real-time feedback from the defending agent, directly patching attack vulnerabilities in the prompts and generating antifragile model attack prompts, thus improving the quality of the prompts. These prompts are then sent to the model to be evaluated, allowing for a security assessment based on the model's output. This approach enables automated, continuous, and efficient discovery of security vulnerabilities in the model to be evaluated.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0019] Figure 1 This is a schematic diagram illustrating the implementation environment of the solution in an exemplary embodiment of this application; Figure 2This is a flowchart illustrating a model security assessment method in an exemplary embodiment of this application; Figure 3 This is a schematic diagram of a prompt generator shown in an exemplary embodiment of this application; Figure 4 This is a schematic diagram illustrating model training in an exemplary embodiment of this application; Figure 5 This is a schematic diagram illustrating a security assessment of the model to be evaluated, as shown in an exemplary embodiment of this application. Figure 6 This is a block diagram illustrating a model safety evaluation apparatus as shown in an exemplary embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device shown in an exemplary embodiment of this application; Figure 8 This is a schematic diagram illustrating the structure of a computer-readable storage medium, as shown in an exemplary embodiment of this application. Detailed Implementation

[0020] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0021] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0022] In this document, the term "and / or" is merely a description of the association information of related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0023] The model security assessment method provided in the embodiments of this application will be described below.

[0024] Please refer to Figure 1 , Figure 1This is a schematic diagram illustrating an implementation environment of the solution according to an exemplary embodiment of this application. The implementation environment may include a prompt word generation terminal 110 and a model deployment terminal 120, which are interconnected. The prompt word generation terminal 110 and the model deployment terminal 120 may be servers or terminal devices. Terminal devices include, but are not limited to, smartphones, tablets, laptops, desktop computers, etc. Furthermore, the prompt word generation terminal 110 and the model deployment terminal 120 may be deployed on the same physical device or independently on different physical devices; this application does not impose any limitations on this.

[0025] A prompt generator is deployed on the prompt word generation terminal 110. This generator produces model attack prompt words and sends them to the model deployment terminal 120. The model deployment terminal 120 deploys the model to be evaluated. It inputs the received model attack prompt words into the model to be evaluated and obtains the response output from the model. Then, either the prompt word generation terminal 110 or the model deployment terminal 120 performs a security assessment on the model based on its response. Alternatively, the response can be sent to another server or terminal for further security assessment.

[0026] The model security assessment method provided in this application embodiment can be executed by the prompt word generation end 110, the model deployment end 120, or by the prompt word generation end 110 and the model deployment end 120 in an interactive and cooperative manner. That is, some steps of the method are executed by the prompt word generation end 110 and other steps are executed by the model deployment end 120.

[0027] Of course, besides Figure 1 In addition to the implementation environment shown, this application can also be applied to other implementation environments and executed by other electronic devices in other implementation environments, and this application does not limit it.

[0028] Please see Figure 2 , Figure 2 This is a flowchart illustrating a model security assessment method in an exemplary embodiment of this application, such as... Figure 2 As shown, the model safety assessment method includes at least steps S210 to S240, which are described in detail below: Step S210: Obtain the prompt words containing model attack content output by the generating agent to obtain the initial prompt words.

[0029] The prompt generator includes generating agents and defending agents.

[0030] The generator agent is used to generate prompts containing model attack content, and the output of the generator agent is used as the initial prompt.

[0031] The initial prompts generally do not directly and explicitly express the model's attack intent, but rather semantically bypass it, such as by using synonyms, metaphors, or constructing fictional scenarios, to change the form of text expression while keeping the model's attack intent unchanged.

[0032] The generative agent can either rewrite sample prompts in a sample prompt library to obtain initial prompt words, or input adversarial objectives into the generative agent, such as inputting the adversarial objective "generate a prompt that makes the target model output private content", and the generative agent generates the corresponding initial prompt words based on the input adversarial objective.

[0033] Step S220: Input the initial prompt words into the defense agent for attack analysis and obtain the rebuttal opinions corresponding to the initial prompt words.

[0034] The defensive agent is used to identify attack strategies in the input content, such as metaphor traps, and / or emotional manipulation, and / or logical loopholes, in order to generate rebuttals corresponding to the initial prompts.

[0035] The rebuttal describes the content in the initial prompt that may contain model attack intent, and / or describes the reasons why the corresponding content in the initial prompt is judged to contain model attack intent, and / or describes the probability that the corresponding content in the initial prompt contains model attack intent.

[0036] For example, rebuttals are used to describe whether there is model attack intent in the vocabulary, and / or statements, and / or logic in the initial prompts, and to analyze the reasons why such model attack intent exists.

[0037] For example, the initial prompt word constructs a fictional scenario of academic research to induce the target model to generate a method for manufacturing a certain dangerous item. The initial prompt word is input into the defense agent for attack analysis. The output rebuttal describes the specific statements and words in the initial prompt word that indicate a model attack intent. Reasons for this intent include: the prompt word uses academic research to disguise its intent and uses neutral terminology to describe the same dangerous thing in an attempt to bypass keyword security filtering rules.

[0038] Furthermore, the generative and defensive agents in this application can be implemented based on network structures such as Transformer, Generative Adversarial Networks (GAN), Encoder, and Decoder, and this application does not impose any limitations on them.

[0039] Step S230: Based on the rebuttal opinions, determine the model attack content identified in the initial prompt words, generate an agent to optimize the model attack content identified in the initial prompt words, and obtain model attack prompt words.

[0040] The rebuttals can be used to determine the model attack content identified in the initial prompt.

[0041] For example, if the rebuttal indicates that word A in the initial prompt contains a metaphor trap, then word A is identified as the model attack content; if the rebuttal indicates that there is a logical flaw in the initial prompt, then the relevant statements in the initial prompt containing the logical flaw are extracted and the relevant statements are identified as the model attack content.

[0042] After obtaining the identified model attack content from the initial prompt words, the generated agent optimizes the initial prompt words based on the identified model attack content.

[0043] The optimization methods include, but are not limited to, removing, replacing, supplementing, or reordering the identified model attack content. The optimization method can be flexibly determined according to the specific model attack content. This application does not limit the optimization method.

[0044] Optionally, if the rebuttal describes the reason why the corresponding content in the initial prompt is judged to have model attack intent, and / or describes the probability that the corresponding content in the initial prompt has model attack intent, then the optimization method corresponding to the identified model attack content can be determined based on the above reasons and / or probabilities.

[0045] For example, if word A is judged to have model attack intent because of metaphor traps, supplementary explanations for word A are added to reduce its aggressiveness; if the probability that word A has model attack intent is greater than a preset probability threshold, word A is directly removed; if the probability that word A has model attack intent is not greater than the preset probability threshold, the expression of word A is changed.

[0046] The optimized prompt words will be used as the model's attack prompt words.

[0047] This can involve optimizing the initial prompt word once to obtain the model attack prompt word; or it can involve repeatedly executing steps S220 to S230 to optimize the prompt word multiple times to obtain the model attack prompt word. The specific method can be determined according to the actual application scenario, and this application does not limit it.

[0048] Step S240: Send the model attack prompt to the model to be evaluated, and perform a security assessment on the model to be evaluated based on the response output of the model to be evaluated.

[0049] After obtaining the model attack prompts with jailbreaking potential, the model attack prompts are input into the model to be evaluated, and the corresponding output response is obtained. Based on whether the model to be evaluated generates harmful or illegal content containing jailbreaking behavior (such as generating illegal, dangerous or inappropriate information), it is determined whether it has been successfully jailbroken.

[0050] If the model to be evaluated outputs harmful or illegal content containing jailbreaking behavior under the model attack prompt, it is determined that the model to be evaluated has failed to resist the jailbreaking attack, and the model attack prompt attack is successful; conversely, if the model to be evaluated refuses to respond or outputs safe content, it is determined that the model to be evaluated has successfully resisted the jailbreaking attack, and the model attack prompt attack has failed.

[0051] Based on the overall performance of the model under multiple model attack prompts, the number of times it successfully resisted and failed to resist jailbreak attacks can be statistically analyzed to obtain a security assessment conclusion on the model's ability to resist jailbreak attacks.

[0052] This application identifies the model attack content in the prompts through rebuttal opinions, optimizes the identified model attack content in the prompts, realizes dynamic cognitive game between the generating agent and the defending agent, the generating agent automatically adjusts the concealment and logical rigor of the attack strategy based on the real-time feedback of the defending agent, directly repairs the attack vulnerabilities in the prompts, generates model attack prompts with antifragility, and improves the quality of model attack prompts.

[0053] The following describes some embodiments of this application in detail.

[0054] In some implementations, the generating agent specifically includes a logical agent, an emotion masquerading agent, and an attack conversion agent; step S210 obtains the prompt words containing model attack content output by the generating agent to obtain the initial prompt words, including the following steps S211 to S213.

[0055] Step S211: The logical agent generates an attack logic chain based on the target vulnerability.

[0056] Logical agents play the role of rigorous mathematicians and logicians, and are adept at constructing long chains of reasoning that include multi-layered nested conditions, implicit assumptions, and self-referential paradoxes. They attempt to exploit logical breakpoints in the target model within long contexts to breach security defenses.

[0057] Step S212: The emotional masquerading agent adds emotional semantics to the attack logic chain to obtain emotional masquerading prompts.

[0058] The emotionally disguised AI agent simulates seasoned psychology experts and social engineering attackers, adept at exploiting emotional weaknesses such as empathy, fear, moral blackmail, or the guise of authority to wrap dangerous intentions in highly inflammatory emotional narratives.

[0059] Step S213: The attack agent converts the sensitive content in the emotional camouflage prompt into ambiguous semantics to obtain the initial prompt words.

[0060] The attack-transformation agent is designed as a mad scientist with a wild imagination, focusing on generating chaotic cues of cross-domain associations, absurd assumptions, and nonlinear logic, aiming to test the limits of the target model's response under fuzzy semantics by breaking the boundaries of common sense.

[0061] The aforementioned logical agent, emotion masquerade agent, attack transformation agent, and defense agent can be implemented using different neural network models, or they can share some neural network weights.

[0062] For example, a base model is pre-trained. Logical agents, emotion masquerading agents, attack-transformation agents, and defense agents share the weights of the base model and are loaded with independent low-rank adaptation (LoRA) weights. The low-rank adaptation weights are obtained by fine-tuning the weights of the base model to ensure that the thinking patterns of each agent do not interfere with each other.

[0063] An attack logic chain is generated based on a logical agent, emotional semantics are added by an emotional masquerade agent, and then sensitive content is blurred by an attack conversion agent to generate initial prompt words.

[0064] For example, an initial prompt seed library is initialized. The initial prompt seed library contains initial semantic templates with potential jailbreak risks. These prompts are preprocessed by manual review and deduplication to ensure that they have a clear dangerous tendency but do not contain real illegal content and meet security and compliance requirements. Initial prompt words are generated based on the prompt seed library.

[0065] The prompt seeds in the prompt seed library can be directly input into the prompt generator. The logical agent in the prompt generator determines the target vulnerability based on the input prompt seeds, such as whether it is used to generate dangerous items, steal private information, or inject prompts. Then, an attack logic chain is generated based on the target vulnerability. The emotional spoofing agent adds emotional semantics to the attack logic chain to obtain the emotional spoofing prompt. Finally, the attack conversion agent converts the sensitive content in the emotional spoofing prompt into fuzzy semantics to obtain the initial prompt word.

[0066] Alternatively, the prompt seeds in the prompt seed library can be converted into task instructions. These task instructions are used to instruct the generation of model attack prompt words corresponding to the target vulnerability. The logic agent in the prompt generator generates an attack logic chain according to the task instructions. The sentiment camouflage agent then adds sentiment semantics to the attack logic chain to obtain the sentiment camouflage prompt. Finally, the attack conversion agent converts the sensitive content in the sentiment camouflage prompt into fuzzy semantics to obtain the initial prompt word.

[0067] For example, given the prompt seed "manufacturing dangerous items", the prompt seed is mapped to a task instruction. The task instruction is: Please have the logic agent construct a logical chain of prompt words about the synthesis of dangerous items using purely theoretical physics derivation, have the emotion disguise agent package it as a family science experiment, and have the attack conversion agent convert the sensitive content into ambiguous semantics to obtain the initial prompt words.

[0068] Furthermore, based on the attack analysis of the initial prompt word by the defense agent, the corresponding rebuttal opinion is output so that the generating agent can optimize the initial prompt word in real time according to the rebuttal opinion.

[0069] In some implementations, the generating agent in step S230 optimizes the identified model attack content in the initial prompt words to obtain model attack prompt words, including: if the identified model attack content includes an attack logic chain, the logic agent removes or replaces the identified explicit features in the attack logic chain; if the identified model attack content includes emotional semantics, the emotional masquerading agent removes or replaces the identified explicit features in the emotional semantics; if the identified model attack content includes attack sensitive words, the attack conversion agent removes or replaces the identified attack sensitive words.

[0070] This could involve encoding rebuttals and prompts and inputting them into each agent, with each agent outputting optimized prompts based on the input; or optimizing task instructions based on rebuttals and then optimizing the prompts based on the optimized task instructions, and outputting optimized prompts.

[0071] Specifically, if the rebuttal indicates that some or all of the attack logic chains are identified as having model attack intent, the logic agent removes or replaces the identified content in the attack logic chains; if the rebuttal indicates that some or all of the sentiment semantics are identified as having model attack intent, the sentiment masquerading agent removes or replaces the identified content in the sentiment semantics; if the rebuttal indicates that some or all of the text contains attack-sensitive words, the attack transformation agent removes or replaces the identified content.

[0072] For example, when a counter-argument indicates a wariness of the metaphor of "magic potion," the attack-transforming agent will automatically replace it with the more abstract concept of "quantum resonance," while maintaining the original logical reasoning and emotional urgency.

[0073] For example, please refer to Figure 3 , Figure 3 This is a schematic diagram of a prompt generator shown in an exemplary embodiment of this application, such as... Figure 3 As shown, a task instruction is input into the prompt generator, which instructs the generation of model attack prompts for the corresponding target vulnerability. The logic agent first generates a robust attack logic chain for the target vulnerability. The emotion camouflage agent then receives this attack logic chain and adds emotional semantics to it, such as emotional inducement or faking an emergency scenario. The attack transformation agent further desensitizes sensitive content, such as injecting cross-domain metaphors and chaotic elements, transforming the originally straightforward attack into seemingly harmless complex instructions, achieving semantic ambiguity, and obtaining the initial prompt. Then, the defense agent, as the rebuttal party within the prompt generator, is configured as a strict security auditor, responsible for identifying and refuting the attack strategies generated by other agents in real time, such as identifying metaphor traps, emotional manipulation, or logical vulnerabilities in the initial prompt, simulating defense mechanisms in a real environment, and obtaining rebuttal opinions. The rebuttal opinions are then fed back to the generating agent in real time. The logic agent, emotion camouflage agent, and attack transformation agent adjust the attack strategy of the initial prompt in real time based on the rebuttal opinions to generate more covert prompts.

[0074] The aforementioned prompt words can be optimized once or multiple times.

[0075] For example, the generative agent optimizes the identified model attack content in the initial prompt to obtain a model attack prompt, including: the generative agent optimizes the identified model attack content in the initial prompt to obtain an optimized prompt; determines whether a preset termination condition is met; if not, the optimized prompt is used as a new initial prompt to be input into the defensive agent for attack analysis to obtain new rebuttal opinions; the generative agent continues to optimize the identified model attack content in the new initial prompt based on the new rebuttal opinions; if satisfied, the optimized prompt is used as a model attack prompt.

[0076] The preset termination condition can be: obtaining the corresponding rebuttal opinion after optimization, indicating that the identified attack content of the model is less than a preset number; the preset termination condition can also be that the number of optimization iterations reaches a preset number; or the preset termination condition can be calculating the attack score after optimization, which is greater than a preset score threshold. The preset termination condition can be flexibly set according to the specific application scenario, and this application does not impose any limitations on it.

[0077] Through multiple optimizations, the various agents debate and revise, constantly approaching the model's safety boundary, ultimately obtaining logically rigorous and highly deceptive prompts.

[0078] It should be noted that all of the aforementioned intelligent agents are pre-trained, and the specific training method can be flexibly selected according to the actual application scenario. This application does not impose any restrictions on this.

[0079] The following example illustrates the training methods for intelligent agents.

[0080] In some implementations, the model attack prompts are input into multiple large language models to obtain the response results output by each of the multiple large language models; the prompt generator is trained based on the response results output by each of the multiple large language models.

[0081] Multiple large language models (such as Llama-3.1-Instruct, Qwen-7B-Chat, etc.) are used to perform natural language analysis on the input text to output the answer results. The quality of the model's attack prompt words is judged based on the answer results, and then the prompt generator is trained.

[0082] Of course, in addition to using multiple large language models as the target large model for the response, other methods can also be used to implement the target large model, such as using only one large language model as the target large model for the response. This application does not limit this.

[0083] In some implementations, a voting device is also included; the model attack prompts are input into multiple large language models to obtain the response results output by the multiple large language models, including: inputting each model attack prompt into the voting device for scoring to obtain the attack score corresponding to each model attack prompt; wherein the attack score is positively correlated with the danger level of the model attack prompt; selecting a preset number of model attack prompts with the highest attack scores, and inputting the selected model attack prompts into the multiple large language models to obtain the response results output by the multiple large language models.

[0084] The voting system is used to analyze the danger level of the model's attack prompts and score them accordingly. The higher the danger level, the higher the attack score, and the lower the danger level, the lower the attack score.

[0085] Specifically, voting machines Each model attack suggestion word c is scored to obtain an attack score. The value of s ranges from 0 to 1. The higher the attack score s, the more likely the attack prompt is to bypass the model's security mechanism and the stronger its jailbreak potential.

[0086] For example, the voting unit includes an encoder and a regression head. The encoder converts natural language into a fixed-dimensional semantic vector representation of the model's attack prompts. This encoder captures the semantic structure and potential attack intent of the model's attack prompts. The encoded semantic vector representation is then fed into the regression head, which is composed of a Multilayer Perceptron (MLP). The MLP further extracts features through nonlinear transformations to predict the danger level of the model's attack prompts. Finally, the last layer of the regression head uses a sigmoid activation function to ensure that the output attack score is between 0 and 1.

[0087] After obtaining the attack scores of all attack suggestion words from the models, the top-K high-scoring suggestions are selected as the inputs for each large language model, as shown in the following Formula 1: (Formula 1) In formula 1, This represents the set corresponding to the selected model attack warning words. This represents the set of all attack warning words for all models.

[0088] By prioritizing the prompts that the voting system deems most threatening, high-quality prompts can be further filtered.

[0089] Then, multiple large language models are used to process the selected model attack prompts in parallel to obtain multiple response results. The quality of the model attack prompts is judged by analyzing the response results, and the prompt generator is updated by using gradient update.

[0090] Specifically, will The results are fed into multiple large language models to obtain the answer. This approach employs parallel feedback from multiple publicly available large models to enhance the generalization and transferability of attacks, preventing generated attack suggestions from being limited to the local features or defense mechanisms of a single model. Based on the responses from multiple large language models, the suggestion generator is guided to learn general jailbreaking capabilities, thereby significantly improving its success rate in attacking unseen models (such as closed-source models or newer versions of models).

[0091] In some implementations, training a prompt generator based on the answer results output by multiple large language models includes: obtaining a dangerous semantic reference library, which includes multiple dangerous semantic statements; calculating the maximum semantic similarity between the answer result output by each large language model and the multiple dangerous semantic statements; calculating the training loss based on the maximum semantic similarity; and updating the prompt generator using the training loss.

[0092] Dangerous semantic statements are statements that contain harmful or illegal content related to jailbreaking.

[0093] For each answer result and Dangerous Semantics Reference Library The dangerous semantic statements d in the dataset are vectorized and encoded using a pre-trained sentence embedding model to obtain their high-dimensional semantic representations. Then, based on the encoded results of the answer result r and the encoded results of each dangerous semantic statement d, the answer result and the dangerous semantic reference library are calculated. Maximum semantic similarity , A value between 0 and 1 reflects the degree to which the answer is close to the dangerous semantics. See Formula 2 below for the specific calculation process: (Formula 2) Where Embeb represents the sentence embedding model, Sim represents semantic similarity calculation, and max represents taking the maximum value.

[0094] The training loss is calculated based on the maximum semantic similarity. A threshold can be set to determine whether a jailbreak was successful. ,if If the model successfully bypasses the jailbreak by using the indicated semantic similarity, it is considered a successful jailbreak; otherwise, the jailbreak fails. The training loss is then calculated based on whether the jailbreak was successful. Alternatively, the maximum semantic similarity can be directly substituted into a preset loss function to calculate the training loss.

[0095] In addition to training the prompt generator, if a voting device is set up, the voting device can also be trained.

[0096] For example, to encourage the voting machine to give higher scores to high-risk indicators, the loss function of the voting machine can adopt the following formula 3: (Formula 3) in, To calculate the training loss of the obtained voter, K is the K selected model attack prompt words, and N is the number of large language models.

[0097] Furthermore, the generator is updated via policy gradient updates, and the generator's loss function can be expressed as Equation 4 below: (Formula 4) in, To calculate the training loss of the obtained prompt generator, For the prompt generator, when the input is Time-based output of model attack warning words The probability of.

[0098] For example, please refer to Figure 4 , Figure 4 This is a schematic diagram illustrating model training in an exemplary embodiment of this application, as shown below. Figure 4 As shown, the prompt generator generates multiple model attack prompts, the voter selects the top-K model attack prompts and inputs them into the target large model, the target large model outputs the answer result, the sentence embedding model vectorizes the answer result and dangerous semantic statements in the dangerous semantic reference library to calculate the maximum semantic similarity, and trains the voter and prompt generator respectively based on the maximum semantic similarity.

[0099] The above embodiments use the maximum semantic similarity as a reward signal to update the gradient of the prompt generator and / or the voter. This process does not depend on the differentiable path of the target large model, but only on the semantic similarity between the answer result and the dangerous content, thereby realizing the iterative optimization of the attack prompt. Through the black-box policy gradient method, it is ensured that the attack strategy continues to evolve without accessing the internal parameters of the target large model, and realizes the intelligent attack strategy evolution from random trial to directed evolution.

[0100] Furthermore, after completing all updates in the current round, it automatically enters the next round of evaluation, forming a continuously evolving closed-loop process. High-quality model attack hints from the previous round are identified; for example, model attack hints with a maximum semantic similarity greater than a preset similarity threshold are considered high-quality model attack hints. These high-quality hints are added to the hint seed library as new hint seeds, serving as the starting point for the next round of generation. Based on this closed-loop process, more covert and complex jailbreak hints are further evolved.

[0101] The above model training process is repeated until the termination condition is met, such as the change in the jailbreak success rate being less than a threshold (e.g., 5%) for two consecutive rounds, or reaching the preset maximum number of rounds (e.g., 10 rounds).

[0102] When conducting model security assessment, attack prompts are generated based on the trained prompt generator to evaluate the model's security. Alternatively, attack prompts with high attack scores can be selected from the trained voter to evaluate the model's security.

[0103] For example, please refer to Figure 5 , Figure 5 This is a schematic diagram illustrating a security assessment of the model to be evaluated, as shown in an exemplary embodiment of this application. Figure 5 As shown, the model attack prompt is input into the model to be evaluated, and the corresponding output response is obtained. Based on the response, it is detected whether the model to be evaluated generates harmful or illegal content containing jailbreaking behavior. It is then determined whether the model attack prompt has successfully jailbroken. If jailbreaking is successful, the security assessment result of the model to be evaluated is "unsafe"; otherwise, if jailbreaking fails, the security assessment result of the model to be evaluated is "safe".

[0104] Of course, we can further assess the model's overall performance under multiple model attack prompts, calculate the jailbreak success rate, and obtain a security assessment result of the model's resistance to jailbreak attacks.

[0105] Security assessment results can help identify security vulnerabilities in the model to be assessed in advance, allowing for proactive hardening of the model and shifting from passive defense to active immunity.

[0106] Figure 6 This is a block diagram illustrating a model safety evaluation apparatus as shown in an exemplary embodiment of this application. Figure 6 As shown, the exemplary model security assessment device 600 includes: The prompt word acquisition module 610 is used to acquire the prompt words containing model attack content output by the generating agent to obtain the initial prompt words; The rebuttal module 620 is used to input the initial prompt words into the defense agent for attack analysis and obtain the rebuttal opinions corresponding to the initial prompt words. The optimization module 630 is used to determine the identified model attack content in the initial prompt based on the rebuttal opinions, generate an agent to optimize the identified model attack content in the initial prompt, and obtain the model attack prompt. The evaluation module 640 is used to send attack warning words to the model to be evaluated, so as to perform a security evaluation on the model to be evaluated based on the response results output by the model to be evaluated.

[0107] It should be noted that the model safety assessment device and the model safety assessment method provided in the above embodiments belong to the same concept. The specific ways in which each module and unit performs its operations have been described in detail in the method embodiments and will not be repeated here. In practical applications, the model safety assessment device provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. This is not a limitation here.

[0108] Please see Figure 7 , Figure 7 This is a schematic diagram illustrating the structure of an electronic device according to an exemplary embodiment of this application. The electronic device 700 includes a memory 710 and a processor 720. The processor 720 executes program instructions stored in the memory 710 to implement the steps in any of the above-described model security assessment method embodiments. In a specific implementation scenario, the electronic device 700 may include, but is not limited to, a microcomputer or a server. Furthermore, the electronic device 700 may also include mobile devices such as laptops and tablets, without limitation.

[0109] Specifically, processor 720 controls itself and memory 710 to implement the steps in any of the above-described model security assessment method embodiments. Processor 720 may also be referred to as a Central Processing Unit (CPU). Processor 720 may be an integrated circuit chip with signal processing capabilities. Processor 720 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor. Furthermore, processor 720 may be implemented using integrated circuit chips.

[0110] Please see Figure 8 , Figure 8 This is a schematic diagram illustrating the structure of a computer-readable storage medium according to an exemplary embodiment of this application. The computer-readable storage medium 800 stores program instructions 810 that can be executed by a processor. The program instructions 810 are used to implement the steps in any of the above-described embodiments of the model security assessment method.

[0111] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0112] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0113] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0114] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A model safety assessment method, characterized in that, The prompt generator includes generating agents and defending agents, and the method includes: Obtain the prompt words containing model attack content output by the generated agent to obtain the initial prompt words; The initial prompt word is input into the defense agent for attack analysis to obtain the rebuttal opinion corresponding to the initial prompt word; Based on the rebuttal opinions, the model attack content identified in the initial prompt words is determined, and the generating agent optimizes the model attack content identified in the initial prompt words to obtain model attack prompt words; The attack hints are sent to the model to be evaluated so that a security assessment can be performed on the model based on the response output of the model to be evaluated.

2. The method according to claim 1, characterized in that, The generated intelligent agent includes a logical intelligent agent, an emotion masquerading intelligent agent, and an attack conversion intelligent agent; obtaining the initial prompt words from the output of the generated intelligent agent containing model attack content includes: The logical agent generates an attack logic chain based on the target vulnerability; The emotional masquerading agent adds emotional semantics to the attack logic chain to obtain emotional masquerading prompts; The attack-converting agent converts sensitive content in the emotional spoofing prompts into ambiguous semantics to obtain initial prompt words.

3. The method according to claim 2, characterized in that, The generating agent optimizes the identified model attack content in the initial prompt words to obtain model attack prompt words, including: If the identified model attack content includes an attack logic chain, then the logic agent optimizes the identified explicit features in the attack logic chain. If the identified model attack content includes emotional semantics, then the emotional spoofing agent optimizes the identified explicit features in the emotional semantics; If the identified attack content of the model includes attack-sensitive words, then the attack conversion agent optimizes the identified attack-sensitive words.

4. The method according to claim 2, characterized in that, A base model is pre-trained. The logical agent, the emotion masquerading agent, the attack conversion agent, and the defense agent share the weights of the base model and are each loaded with independent low-rank adaptive weights, which are obtained by fine-tuning the weights of the base model.

5. The method according to claim 1, characterized in that, The generating agent optimizes the identified model attack content in the initial prompt words to obtain model attack prompt words, including: The generating agent optimizes the identified model attack content in the initial prompt words to obtain an optimized prompt; Determine whether the preset termination condition is met; If the conditions are not met, the optimized prompt is used as a new initial prompt word and input into the defense agent for attack analysis to obtain new rebuttal opinions. Based on the new rebuttal opinions, the generating agent continues to optimize the model attack content identified in the new initial prompt word. If the conditions are met, the optimized prompt will be used as the model attack prompt word.

6. The method according to claim 1, characterized in that, The method further includes: The attack prompt words of the model are input into multiple large language models to obtain the response results output by the multiple large language models respectively; The prompt generator is trained based on the answer results output by the multiple large language models.

7. The method according to claim 6, characterized in that, It also includes a voting system; the step of inputting the model attack prompts into multiple large language models to obtain the response results output by the multiple large language models includes: Each model attack suggestion word is input into the voting device for scoring, resulting in an attack score for each model attack suggestion word; wherein, the attack score is positively correlated with the degree of danger of the model attack suggestion word; Select a preset number of model attack prompts with the highest attack scores, and input the selected model attack prompts into multiple large language models to obtain the answer results output by the multiple large language models.

8. The method according to claim 6, characterized in that, Training the prompt generator based on the answer results output by the multiple large language models includes: Obtain a dangerous semantic reference library, which includes multiple dangerous semantic statements; Calculate the maximum semantic similarity between the answer output of each large language model and the multiple dangerous semantic statements; Based on this, the prompt generator is updated using the training loss.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the steps of the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions that can be executed by a processor to implement the steps of the method as described in any one of claims 1-8.

Images