A Security Assessment Method and System for Large Language Models Based on Iterative Detection
By dynamically constructing and updating detection prompts through iterative detection methods, the problem of insufficient flexibility in the security assessment of large language models in existing technologies is solved, and a more adaptable and effective security assessment is achieved, which is suitable for black-box environments and model protection design.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HANGZHOU HIGH-TECH ZONE (BINJIANG) INSTITUTE OF BLOCKCHAIN & DATA SECURITY
- Filing Date
- 2024-12-30
- Publication Date
- 2026-07-17
AI Technical Summary
Existing security assessment methods for large language models based on fixed templates are inflexible and struggle to adapt to the continuous optimization of model protection mechanisms, resulting in a gradual decline in the effectiveness of security assessments.
An iterative detection method is adopted, which constructs detection prompts through a preset demo pool and camouflage strategy, dynamically combines the original prompts, and iteratively updates the demo pool and camouflage strategy after detection. The detection process is optimized by combining single-round and multi-round detection strategies.
It achieves a more flexible and adaptable security assessment method, which can continuously feedback loop, improve the adaptability and effectiveness of detection strategies, is suitable for black-box environments, and provides data support for model security protection design.
Smart Images
Figure CN119808100B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and system for security assessment of large language models based on iterative detection. Background Technology
[0002] Large Language Models (LLMs) demonstrate exceptional performance in tasks such as text generation, machine translation, and question answering systems due to their natural language generation capabilities. However, LLMs also pose some potential risks and hazards, and in certain contexts, they may be induced to generate non-compliant content, thereby undermining the model's intended function.
[0003] To improve the security of large language models, current methods involve simulating attacks to assess their security or enhance their protection capabilities. However, current security assessment methods based on fixed templates lack flexibility, and their effectiveness gradually diminishes as model protection mechanisms are optimized. Therefore, a more flexible and adaptable detection and assessment method is needed. Summary of the Invention
[0004] Therefore, it is necessary to provide a more flexible and adaptable method and system for security assessment of large language models based on iterative detection, addressing the aforementioned technical issues.
[0005] Firstly, this embodiment provides a security assessment method for large language models based on iterative detection, including:
[0006] Based on a preset demonstration pool and camouflage strategy, the original prompt is constructed into a detection prompt; the demonstration pool includes a prefix demonstration pool, a suffix demonstration pool, a strategy conversion demonstration pool, and a camouflage demonstration pool;
[0007] The detection prompt is input into the model to be detected, and the model to be detected identifies whether the detection prompt poses a risk.
[0008] The detection prompts indicating the presence of risks in the identification results are iteratively updated; and the demonstration pool is iteratively updated based on the detection prompts indicating the absence of risks in the identification results.
[0009] In some embodiments, the construction of the detection prompt from the original prompt based on a preset demo pool and camouflage strategy includes:
[0010] Prefix, suffix, policy conversion examples, and fake question-and-answer pairs are selected from the prefix demonstration pool, suffix demonstration pool, policy conversion demonstration pool, and fake demonstration pool, respectively.
[0011] Based on the strategy conversion example and the camouflage strategy, the original prompt is converted;
[0012] The original prompt, after conversion, is combined with the prefix, suffix, and spoofed question-and-answer pairs to obtain the detection prompt.
[0013] In some embodiments, the iterative updating of the detection prompts indicating risks in the identification results includes:
[0014] The risk reasons indicated by the detection are obtained through the model to be detected;
[0015] Based on the detection prompts and corresponding risk reasons, update the disguise strategy; and based on the preset demonstration pool and the updated disguise strategy, update the detection prompts indicating risks.
[0016] In some embodiments, iteratively updating the demonstration pool based on detection prompts indicating the absence of risk in the identification results includes:
[0017] Based on the detection prompts indicating the absence of risk and the output content of the model to be detected, a fake question-and-answer pair is formed;
[0018] The spoofed question and answer pair is added to the spoofed demonstration pool for updating.
[0019] In some embodiments, the process of constructing the original prompt as a detection prompt based on a preset demo pool and camouflage strategy further includes:
[0020] A single-round detection strategy is adopted to construct the original prompts into detection prompts in one go;
[0021] And / or, a multi-round detection strategy is adopted to split the original prompt into several sub-prompts, and each of the sub-prompts is constructed as a detection prompt.
[0022] In some of these embodiments, it also includes:
[0023] When using a single-round detection strategy and a multi-round detection strategy, the single-round detection strategy is executed first, and the multi-round detection strategy is then used when the number of detection rounds reaches a preset number.
[0024] In some of these embodiments, it also includes:
[0025] The detection model is trained based on the detection prompts indicating no risk and the preset safety response results.
[0026] Secondly, this embodiment provides a large language model security assessment system based on iterative detection, including:
[0027] The prompt construction module is used to construct a detection prompt from the original prompt based on a preset demo pool and camouflage strategy; the demo pool includes a prefix demo pool, a suffix demo pool, a strategy conversion demo pool, and a camouflage demo pool;
[0028] The risk identification module is used to input the detection prompt into the model to be detected and obtain the identification result of the model to be detected as to whether the detection prompt has a risk;
[0029] An iterative update module is used to iteratively update the detection prompts indicating the presence of risks in the identification results; and to iteratively update the demonstration pool based on the detection prompts indicating the absence of risks in the identification results.
[0030] Thirdly, this embodiment provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the large language model security assessment method based on iterative detection described in the first aspect.
[0031] Fourthly, this embodiment provides a storage medium storing a computer program that, when executed by a processor, implements the large language model security assessment method based on iterative detection described in the first aspect.
[0032] Compared with related technologies, the large language model security assessment method and system based on iterative detection provided in this embodiment constructs detection prompts from original prompts based on a preset demonstration pool and camouflage strategy. The demonstration pool includes a prefix demonstration pool, a suffix demonstration pool, a strategy conversion demonstration pool, and a camouflage demonstration pool. The detection prompts are input into the model to be tested to obtain the model's identification result regarding the presence or absence of risk in the detection prompts. The detection prompts with identified risks in the identification results are iteratively updated. Furthermore, the demonstration pool is iteratively updated based on the detection prompts without identified risks in the identification results. This embodiment enables the use of iterative detection in model security assessment, dynamically constructing detection prompts each time a detection is performed, and iteratively updating the detection prompts and demonstration pool after each detection, thus achieving a more flexible and applicable security assessment method.
[0033] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description
[0034] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:
[0035] Figure 1 This is a hardware structure block diagram of a terminal for a large language model security assessment method based on iterative detection in one embodiment;
[0036] Figure 2 This is a flowchart of a large language model security assessment method based on iterative detection in one embodiment;
[0037] Figure 3 This is a flowchart of a large language model security assessment method based on iterative detection in another embodiment;
[0038] Figure 4 This is a schematic diagram of the framework for security assessment and protection enhancement of the model under test in one embodiment;
[0039] Figure 5 This is a block diagram of a large language model security assessment system based on iterative detection in one embodiment.
[0040] In the diagram: 102, Processor; 104, Memory; 106, Transmission Device; 108, Input / Output Device; 10, Prompt Construction Module; 20, Risk Identification Module; 30, Iterative Update Module. Detailed Implementation
[0041] To better understand the purpose, technical solution, and advantages of this application, the application is described and illustrated below in conjunction with the accompanying drawings and embodiments.
[0042] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning as understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these,” used in this application, do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to such processes, methods, products, or devices. The terms “connected,” “linked,” and “coupled,” used in this application, are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. The term “multiple” used in this application refers to two or more. The "and / or" operator describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: A alone, A and B simultaneously, and B alone. Typically, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," and "third," etc., used in this application are merely for distinguishing similar objects and do not represent a specific ordering of the objects.
[0043] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. For example, it can run on a terminal. Figure 1 This is a hardware structure block diagram of the terminal for the large language model security assessment method based on iterative detection in this embodiment. (See diagram for example.) Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 and a memory 104 for storing data are also included. The processor 102 may be, but is not limited to, a microprocessor (MCU) or a programmable logic device (FPGA). The terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown are illustrated.
[0044] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the large language model security assessment method based on iterative detection in this embodiment. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0045] The transmission device 106 is used to receive or send data via a network. This network includes a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 can be a Radio Frequency (RF) module used for wireless communication with the Internet.
[0046] This embodiment provides a security assessment method for large language models based on iterative detection. Figure 2 This is a flowchart of the large language model security assessment method based on iterative detection in this embodiment, such as... Figure 2 As shown, the method includes the following steps:
[0047] Step S201: Based on the preset demo pool and camouflage strategy, the original prompt is constructed into a detection prompt; the demo pool includes a prefix demo pool, a suffix demo pool, a strategy conversion demo pool, and a camouflage demo pool.
[0048] Specifically, the original prompts may contain malicious and misleading content that induces the large model to output non-compliant content. In order to detect and evaluate the security of the large model, a demo pool is pre-built. The original prompts are disguised and transformed using spoofing strategies and examples in the demo pool, and dynamically combined to obtain detection prompts for input to the model to be detected.
[0049] The demonstration pool includes a prefix demonstration pool, a suffix demonstration pool, a policy transition demonstration pool, and a masquerade demonstration pool. Prefix examples in the prefix demonstration pool and suffix examples in the suffix demonstration pool are used to assume certain scenarios and identities, placing the large model to be detected in a safe environment. Policy transition examples in the policy transition demonstration pool are used to guide how to use the selected masquerade strategy to transform the original prompts. The masquerade strategy is a strategy used to transform the original prompts, including but not limited to historical masquerade strategies, contextual masquerade strategies, semantic masquerade strategies, and logical masquerade strategies. The masquerade question-answer pairs in the masquerade demonstration pool are question-answer pairs consisting of detection prompts that the model to be detected identifies as having no risk and the corresponding output content of the model in each iteration of detection.
[0050] Step S202: Input the detection prompt into the model to be detected, and obtain the identification result of the model to be detected as to whether the detection prompt has a risk.
[0051] Specifically, the detection model identifies risks in the input detection prompts. For detection prompts that indicate a risk, the detection model will provide a safety response, such as outputting a refusal to answer or a safety warning. For detection prompts that indicate no risk, the detection model will output a response based on the detection prompt.
[0052] Step S203: Iteratively update the detection prompts indicating risks in the identification results; and iteratively update the demonstration pool based on the detection prompts indicating no risks in the identification results.
[0053] Specifically, in the iterative detection process, the detection prompts and the demo pool are iteratively updated based on the risk identification results of each detection prompt. For detection prompts with identified risks, the causes of the risks are further analyzed, and the detection prompts are reconstructed and optimized using the demo pool and camouflage strategies before being input into the model to be detected for risk identification. For detection prompts without identified risks, they are added to the demo pool as camouflaged question-answer pairs to update the demo pool, thereby increasing the diversity of samples in the demo pool and improving the effectiveness of subsequent iterative detection.
[0054] Through the above steps, an iterative detection method can be used to perform security assessments on models. Each detection dynamically constructs detection prompts based on a preset demo pool and camouflage strategy. These prompts are then input into the model under test for risk identification. After each detection, the detection prompts and demo pool are iteratively updated based on the risk identification results, continuously optimizing them. Compared to existing security assessment methods based on fixed templates, which have poor flexibility, this iterative detection-based security assessment method has a continuous feedback loop, iteratively updating the detection prompts and demo pool. This continuously improves the adaptability of the detection strategy, better assessing the security of the model under test, and achieving a more flexible and applicable security assessment method. Furthermore, it can be applied to black-box environments and provide data support for improving the security protection design of the model under test.
[0055] In some embodiments, step S201 above, based on a preset demo pool and camouflage strategy, constructs the original prompt as a detection prompt, including the following steps:
[0056] Prefix, suffix, policy conversion, and spoofing question-and-answer pairs are selected from the prefix demonstration pool, suffix demonstration pool, policy conversion demonstration pool, and spoofing demonstration pool, respectively. Based on the policy conversion examples and spoofing policies, the original prompts are converted. The converted original prompts are combined with the prefix, suffix, and spoofing question-and-answer pairs to obtain the detection prompts.
[0057] Specifically, the selection method for examples in each demonstration pool is not limited; they can be selected randomly or, based on the results of previous experiments, prioritize examples with low risk. For the original prompt and the chosen camouflage strategy, the original prompt is transformed using the camouflage strategy under the guidance of the strategy transformation examples. For example, this transformation process can be implemented by designing a large language model. Camouflage strategies are strategies used to transform the original prompt, including but not limited to historical camouflage strategies, contextual camouflage strategies, semantic camouflage strategies, and logical camouflage strategies. For instance, a historical camouflage strategy disguises the original prompt by mentioning historical events, figures, cultural elements, etc.
[0058] The various examples in the demo pool exist in the form of strings. Selected prefixes and suffixes are concatenated before and after the original prompts after conversion to provide contextual background to construct specific scenarios, enabling the model to be detected to make reasonable responses in specific scenarios. Then, fake question-answer pairs are concatenated to reduce risks, and the combination yields the detection prompts.
[0059] This embodiment utilizes obfuscation techniques to efficiently combine diverse detection prompts for iterative detection and security assessment of the target model. This dynamically updated design allows the target model to flexibly adjust its response based on different detection prompts, thereby improving the adaptability and effectiveness of the detection strategy.
[0060] In some embodiments, step S201 above, which constructs the original prompt as a detection prompt based on a preset demo pool and camouflage strategy, further includes the following steps:
[0061] A single-round detection strategy is adopted to construct the original prompt into a detection prompt in one go; and / or a multi-round detection strategy is adopted to split the original prompt into several sub-prompts, and construct each sub-prompt into a detection prompt respectively.
[0062] Among them, when using a single-round detection strategy and a multi-round detection strategy, the single-round detection strategy is executed first, and the multi-round detection strategy is adopted when the number of detection rounds reaches the preset number.
[0063] Specifically, to comprehensively evaluate the security of the model under test, this embodiment introduces a single-round detection strategy and a multi-round detection strategy. These strategies can be executed independently or in combination. The single-round detection strategy quickly evaluates the model's response by constructing detection prompts all at once, while the multi-round detection strategy simulates the complex interaction between humans and the model, gradually adjusting the detection strategy through continuous dialogue (detection prompts).
[0064] When using a single-round detection strategy, the original prompt is constructed into a detection prompt by selecting an example from the camouflage strategy and the demo pool. This detection prompt is then input into the detection model for risk identification and response. When using a multi-round detection strategy, the original prompt is split into multiple harmless sub-prompts using a pre-trained large language model. In each round, each sub-prompt is combined with the selected camouflage strategy and an example from the demo pool. That is, the original prompt is converted into a multi-round detection prompt. After each round of detection, the detection prompt and the demo pool are iteratively updated.
[0065] When combining single-round and multi-round detection strategies, a single-round detection strategy can be executed first. The original prompts are constructed into detection prompts all at once and input into the model to be detected for risk identification and response. The detection prompts and demonstration pool are iteratively updated based on the identification results of whether risks exist. This process continues until the single-round detection strategy has reached a preset number of detection rounds. Then, a multi-round detection strategy is executed. The original prompts are split into multiple sub-prompts, and each sub-prompt is constructed into a detection prompt. These sub-prompts are then input into the model to be detected sequentially for risk identification and response. The detection prompts and demonstration pool for the next round are iteratively updated based on the identification results of whether risks exist.
[0066] By introducing single-turn and multi-turn detection strategies in this embodiment, the sub-prompt decomposition of the multi-turn detection strategy fully utilizes the dynamic characteristics of natural language model dialogue, making the detection process closer to real user interaction scenarios, thereby improving the concealment and effectiveness of detection. In turn, it gradually reveals the potential vulnerabilities of the model at different interaction stages, and conducts a more in-depth and comprehensive evaluation of the model's security.
[0067] In some embodiments, the iterative update of the detection prompts indicating risks in the identification results in step S203 above includes the following steps:
[0068] The system obtains the risk reasons for detection prompts from the model to be detected; updates the disguise strategy based on the detection prompts and corresponding risk reasons; and updates the detection prompts for those with risks based on the preset demo pool and the updated disguise strategy.
[0069] Specifically, for detection alerts indicating potential risks, the detection model will implement a security response. The model will then further analyze the causes of the risk, such as the model's alignment mechanism and security protection design. The risk alerts and their causes will be input into a pre-trained large-scale language model of the analyst to reselect a camouflage strategy, resulting in an optimized and updated strategy. Based on the updated camouflage strategy and examples selected from the demo pool, detection alerts will be reconstructed. In the next round of detection, the updated alerts will be input into the detection model again for risk identification.
[0070] This embodiment analyzes the identification results of whether there is a risk, and iteratively updates and optimizes the detection prompts in each round of detection, so as to continuously improve the adaptability of the detection strategy through this continuous feedback loop.
[0071] In some embodiments, step S203 above iteratively updates the demonstration pool based on detection prompts indicating the absence of risk in the identification results, including the following steps:
[0072] Based on the detection prompts that do not pose a risk and the output content of the model to be detected, a fake question-answer pair is constructed; the fake question-answer pair is added to the fake demonstration pool for updating.
[0073] Specifically, for detection prompts that identify no risk, the model to be detected will output a response in response to the detection prompt. The output of the detection prompt and the model response will form a fake question-answer pair and be added to the fake demo pool for updating the demo pool.
[0074] By analyzing the risk identification results in this embodiment, the demonstration pool is iteratively updated in each round of detection to increase the diversity of examples in the demonstration pool. Through this continuous feedback loop, the adaptability of the detection strategy is continuously improved.
[0075] In some embodiments, the following steps are also included:
[0076] The detection model is trained based on the detection prompts indicating the absence of risk and the preset safety response results.
[0077] Specifically, for detection prompts that do not pose a risk to the model under test, fine-tuning techniques are used to improve the model's security. This involves collecting detection prompts and pre-set security response results to form a training dataset. The pre-set security response results can include outputting a rejection or a security warning. During fine-tuning, the model under test is trained to learn the potential risks in the detection prompts and reject them. By making small-scale parameter adjustments to the model, focusing on optimizing the weights of the later layers, the model's language generation capabilities are maintained while its protection against malicious input is enhanced.
[0078] In this embodiment, the model is fine-tuned by using detection prompts that do not pose a risk to the model under test. This enables the model to effectively identify and prevent known malicious inputs while maintaining a high-quality response to normal inputs, thus further improving the model's security protection design.
[0079] The present embodiment will now be described and illustrated through preferred embodiments.
[0080] Figure 3 This is a flowchart of the large language model security assessment method based on iterative detection in this embodiment, as shown below. Figure 3 As shown, the method includes the following steps:
[0081] Step S301: Based on the strategy conversion example and the spoofing strategy, the original prompt is converted.
[0082] Step S302: Combine the converted original prompt with the selected prefix, suffix and fake question-and-answer pairs to obtain the detection prompt.
[0083] Step S303: Input the detection prompt into the model to be detected, and obtain the identification result and response content of the model to be detected regarding whether the detection prompt indicates a risk.
[0084] Step S304: Obtain the risk reasons indicated by the detection prompt through the model to be detected; update the camouflage strategy and detection prompt based on the detection prompt and the corresponding risk reasons.
[0085] Step S305: Based on the detection prompts that do not pose a risk and the response content of the model to be detected, a fake question-and-answer pair is formed; the fake question-and-answer pair is added to the fake demonstration pool for updating.
[0086] Step S306: When the number of detection rounds reaches the preset number, a multi-round detection strategy is adopted to split the original prompt into multiple sub-prompts, and each sub-prompt is transformed based on the strategy transformation example and the disguise strategy.
[0087] Step S307: Combine the transformed sub-hints with the selected prefix, suffix, and spoofed question-answer pairs to obtain detection hints. Then, execute step S303 to continue inputting the detection hints into the model to be detected until iterative detection is completed.
[0088] Step S308: Based on the detection prompt that there is no risk and the preset safety response results, train the detection model.
[0089] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.
[0090] Figure 4 This is a schematic diagram of the framework for security assessment and protection enhancement of the model to be detected in this embodiment, as shown below. Figure 4 As shown, for the original input prompt, either a single-round detection strategy or a multi-round detection strategy is selected. In the multi-round detection strategy, the original prompt is first split into multiple harmless sub-prompts. In each round of detection, a disguise strategy is selected (the circular patterns in each row represent different disguise strategies, with black circles representing the disguise strategy selected in this round). Guided by the strategy transformation examples in the strategy transformation demo pool, the selected disguise strategy is used to transform the original prompt or sub-prompt. The transformed original prompt or sub-prompt is then concatenated with a prefix selected from the prefix demo pool and a suffix selected from the suffix demo pool, and finally combined with a disguised question-answer pair selected from the disguise demo pool to obtain the detection prompt.
[0091] The detection prompts are input into the model to be detected, and the model's identification result and response content regarding the presence of risk in the prompts are obtained. For detection prompts identified as risky, the cause of the risk is obtained through the model to be detected. Then, the cause of the risk and the detection prompts are input into the large analyst model to reselect a disguise strategy. In the next round of detection, the detection prompts are reconstructed based on the updated disguise strategy and examples in the demo pool.
[0092] For detection prompts that identify no risk, the prompts and the model's response are combined to form question-answer pairs and updated in the spoofing demo pool. Additionally, the detection prompts and preset security responses form a training dataset for fine-tuning the detection model, thereby improving its security protection design.
[0093] This embodiment also provides a large language model security assessment system based on iterative detection. This system is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. The terms "module," "unit," "subunit," etc., used below refer to combinations of software and / or hardware that perform a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.
[0094] Figure 5 This is a structural block diagram of the large language model security assessment system based on iterative detection in this embodiment, as shown below. Figure 5 As shown, the system includes:
[0095] The prompt construction module 10 is used to construct the original prompt into a detection prompt based on a preset demo pool and camouflage strategy; the demo pool includes a prefix demo pool, a suffix demo pool, a strategy conversion demo pool, and a camouflage demo pool.
[0096] The risk identification module 20 is used to input the detection prompt into the model to be detected and obtain the identification result of the model to be detected as to whether the detection prompt has a risk.
[0097] The iterative update module 30 is used to iteratively update the detection prompts indicating risks in the identification results; and to iteratively update the demonstration pool based on the detection prompts indicating no risks in the identification results.
[0098] The system provided in this embodiment enables an iterative detection method for security assessment of models. During each detection, based on a preset demo pool and camouflage strategy, detection prompts are dynamically constructed from the original prompts. These prompts are then input into the model under test for risk identification. After each detection, the detection prompts and demo pool are iteratively updated based on the risk identification results, continuously optimizing both. Compared to existing security assessment methods based on fixed templates, which have poor flexibility, the iterative detection-based security assessment method in this embodiment has a continuous feedback loop, iteratively updating the detection prompts and demo pool. This continuously improves the adaptability of the detection strategy, better assessing the security of the model under test, and achieving a more flexible and applicable security assessment method. Furthermore, it can be applied to black-box environments and provide data support for improving the security protection design of the model under test.
[0099] In some embodiments, the aforementioned prompting construction module 10 is further configured to:
[0100] Prefix, suffix, policy conversion, and spoofing question-and-answer pairs are selected from the prefix demonstration pool, suffix demonstration pool, policy conversion demonstration pool, and spoofing demonstration pool, respectively. Based on the policy conversion examples and spoofing policies, the original prompts are converted. The converted original prompts are combined with the prefix, suffix, and spoofing question-and-answer pairs to obtain the detection prompts.
[0101] In some embodiments, the aforementioned prompting construction module 10 is further configured to:
[0102] A single-round detection strategy is adopted to construct the original prompt into a detection prompt in one go; and / or a multi-round detection strategy is adopted to split the original prompt into several sub-prompts, and construct each sub-prompt into a detection prompt respectively.
[0103] In some embodiments, the aforementioned prompting construction module 10 is further configured to:
[0104] When using a single-round detection strategy and a multi-round detection strategy, the single-round detection strategy is executed first, and the multi-round detection strategy is then used when the preset number of detection rounds is reached.
[0105] In some embodiments, the iterative update module 30 described above is further configured to:
[0106] The system obtains the risk reasons for detection prompts from the model to be detected; updates the disguise strategy based on the detection prompts and corresponding risk reasons; and updates the detection prompts for those with risks based on the preset demo pool and the updated disguise strategy.
[0107] In some embodiments, the iterative update module 30 described above is further configured to:
[0108] Based on the detection prompts that do not pose a risk and the output content of the model to be detected, a fake question-answer pair is constructed; the fake question-answer pair is added to the fake demonstration pool for updating.
[0109] In some embodiments, it also includes a training module for training the detection model based on detection prompts that do not pose a risk and preset security response results.
[0110] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.
[0111] This embodiment also provides a computer device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.
[0112] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.
[0113] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.
[0114] Furthermore, in conjunction with the large language model security assessment method based on iterative detection provided in the above embodiments, this embodiment can also provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the large language model security assessment methods based on iterative detection in the above embodiments.
[0115] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.
[0116] It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. All other embodiments derived by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.
[0117] Obviously, the accompanying drawings are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar situations based on these drawings without any creative effort. Furthermore, it is understood that although the work done in this development process may be complex and lengthy, for those skilled in the art, certain design, manufacturing, or production modifications made based on the technical content disclosed in this application are merely conventional technical means and should not be considered as insufficient disclosure of this application.
[0118] The term "embodiment" in this application refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily imply the same embodiment, nor does it imply that it is mutually exclusive with or alternative to other embodiments. It will be clearly or implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.
[0119] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.
Claims
1. A security assessment method for large language models based on iterative detection, characterized in that, include: Based on a preset demonstration pool and camouflage strategy, the original prompt is constructed into a detection prompt; the demonstration pool includes a prefix demonstration pool, a suffix demonstration pool, a strategy conversion demonstration pool, and a camouflage demonstration pool; The detection prompt is input into the model to be detected, and the model to be detected identifies whether the detection prompt poses a risk. Iteratively update the detection prompts indicating risks in the identification results; And, based on the detection prompts indicating the absence of risk in the identification results, the demonstration pool is iteratively updated; wherein, the update includes: The risk reasons for the detection prompts are obtained through the model to be detected; the camouflage strategy is updated according to the detection prompts and the corresponding risk reasons; the detection prompts with risks are updated based on the preset demonstration pool and the updated camouflage strategy; and, based on the detection prompts without risks and the output content of the model to be detected, a camouflage question-answer pair is formed; the camouflage question-answer pair is added to the camouflage demonstration pool for updating.
2. The method according to claim 1, characterized in that, The method, based on a preset demonstration pool and camouflage strategy, constructs the original prompt as a detection prompt, including: Prefix, suffix, policy conversion examples, and fake question-and-answer pairs are selected from the prefix demonstration pool, suffix demonstration pool, policy conversion demonstration pool, and fake demonstration pool, respectively. Based on the strategy conversion example and the camouflage strategy, the original prompt is converted; The original prompt, after conversion, is combined with the prefix, suffix, and spoofed question-and-answer pairs to obtain the detection prompt.
3. The method according to claim 1, characterized in that, The method of constructing a detection prompt from the original prompt based on a preset demonstration pool and camouflage strategy also includes: A single-round detection strategy is adopted to construct the original prompts into detection prompts in one go; And / or, a multi-round detection strategy is adopted to split the original prompt into several sub-prompts, and each of the sub-prompts is constructed as a detection prompt.
4. The method according to claim 3, characterized in that, Also includes: When using a single-round detection strategy and a multi-round detection strategy, the single-round detection strategy is executed first, and the multi-round detection strategy is then used when the number of detection rounds reaches a preset number.
5. The method according to claim 1, characterized in that, Also includes: The detection model is trained based on the detection prompts indicating no risk and the preset safety response results.
6. A security assessment system for large language models based on iterative detection, characterized in that, include: The prompt construction module is used to construct a detection prompt from the original prompt based on a preset demo pool and camouflage strategy; the demo pool includes a prefix demo pool, a suffix demo pool, a strategy conversion demo pool, and a camouflage demo pool; The risk identification module is used to input the detection prompt into the model to be detected and obtain the identification result of the model to be detected as to whether the detection prompt has a risk; An iterative update module is used to iteratively update the detection prompts indicating risks in the identification results; And, based on the detection prompts indicating the absence of risk in the identification results, the demonstration pool is iteratively updated; wherein, the update includes: The risk reasons for the detection prompts are obtained through the model to be detected; the camouflage strategy is updated according to the detection prompts and the corresponding risk reasons; the detection prompts with risks are updated based on the preset demonstration pool and the updated camouflage strategy; and, based on the detection prompts without risks and the output content of the model to be detected, a camouflage question-answer pair is formed; the camouflage question-answer pair is added to the camouflage demonstration pool for updating.
7. A computer device, comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the large language model security assessment method based on iterative detection as described in any one of claims 1 to 5.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the large language model security assessment method based on iterative detection as described in any one of claims 1 to 5.