Large language model security assessment method and system based on iterative detection

Abstract

The invention relates to a large language model safety assessment method and system based on iterative detection, and the method comprises the steps: constructing an original prompt into a detection prompt based on a preset demonstration pool and a camouflage strategy; the demonstration pool comprises a prefix demonstration pool, a suffix demonstration pool, a strategy conversion demonstration pool and a disguise demonstration pool; inputting the detection prompt into the to-be-detected model to obtain an identification result of the to-be-detected model on whether the detection prompt has a risk; iteratively updating detection prompts with risks in the identification result; and iteratively updating the demonstration pool according to the detection prompt that no risk exists in the identification result. Through the method and the device, an iterative detection method can be adopted when the model is subjected to safety assessment, the detection prompt is dynamically constructed during each detection, and the detection prompt and the demonstration pool are iteratively updated after detection, so that the safety assessment method which is more flexible and higher in applicability is realized.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a large language model security assessment method and system based on iterative detection. Background Art

[0002] Large Language Models (LLMs) have demonstrated excellent performance in tasks such as text generation, machine translation, and question-answering systems through their natural language generation capabilities. However, large language models also have some potential hazards and risks. In certain situations, they may be induced to generate non-compliant content, destroying the intended function of the model.

[0003] In order to improve the security of large language models, we can currently conduct security assessments on large models or improve the security protection capabilities of models by simulating attacks. However, the current security assessment methods based on fixed templates are less flexible, and as the model protection mechanism is continuously optimized, the effect will gradually deteriorate. Therefore, a more flexible and adaptable detection and assessment method is needed. Summary of the invention

[0004] Based on this, it is necessary to provide a more flexible and adaptable large language model security assessment method and system based on iterative detection to address the above technical problems.

[0005] In the first aspect, a large language model security assessment method based on iterative detection is provided in this embodiment, including:

[0006] Based on a preset demonstration pool and camouflage strategy, the original prompt is constructed into a detection prompt; the demonstration pool includes a prefix demonstration pool, a suffix demonstration pool, a strategy conversion demonstration pool and a camouflage demonstration pool;

[0007] Inputting the detection prompt into the model to be detected, and obtaining a recognition result of the model to be detected on whether the detection prompt has a risk;

[0008] Iteratively updating detection prompts for risks in the recognition results; and iteratively updating the demonstration pool based on detection prompts for risks not existing in the recognition results.

[0009] In some of the embodiments, the method of constructing the original prompt into a detection prompt based on a preset demonstration pool and camouflage strategy includes:

[0010] Select prefixes, suffixes, strategy conversion examples and disguised question-answer pairs from the prefix demonstration pool, suffix demonstration pool, strategy conversion demonstration pool and disguised demonstration pool respectively;

[0011] Based on the strategy conversion example and the disguise strategy, convert the original prompt;

[0012] The converted original prompt is combined with the prefix, suffix and disguised question-answer pair to obtain the detection prompt.

[0013] In some of the embodiments, the iterative updating of the detection prompt of the risk in the recognition result includes:

[0014] Obtaining the risk reason prompted by the detection through the model to be detected;

[0015] According to the detection prompt and the corresponding risk reason, the camouflage strategy is updated; and based on the preset demonstration pool and the updated camouflage strategy, the detection prompt of the risk is updated.

[0016] In some of the embodiments, iteratively updating the demonstration pool according to the detection prompt that there is no risk in the recognition result includes:

[0017] Based on the detection prompt that there is no risk and the output content of the model to be detected, a disguised question-answer pair is formed;

[0018] The disguised question and answer pair is added to the disguised demonstration pool for updating.

[0019] In some of the embodiments, the process of constructing the original prompt as a detection prompt based on a preset demonstration pool and camouflage strategy further includes:

[0020] A single-round detection strategy is adopted to construct the original prompt into a detection prompt at one time;

[0021] And / or, a multi-round detection strategy is adopted to split the original prompt into a plurality of sub-prompts, and each of the sub-prompts is constructed as a detection prompt.

[0022] In some of the embodiments, it also includes:

[0023] When a single-round detection strategy and a multi-round detection strategy are adopted, the single-round detection strategy is first executed until the number of detection rounds reaches a preset number, and then the multi-round detection strategy is adopted.

[0024] In some of the embodiments, it also includes:

[0025] Based on the detection prompt that there is no risk and the preset safety response result, the model to be detected is trained.

[0026] In a second aspect, in this embodiment, a large language model security assessment system based on iterative detection is provided, including:

[0027] A prompt construction module is used to construct the original prompt into a detection prompt based on a preset demonstration pool and a disguise strategy; the demonstration pool includes a prefix demonstration pool, a suffix demonstration pool, a strategy conversion demonstration pool and a disguise demonstration pool;

[0028] A risk identification module, used for inputting the detection prompt into the model to be detected, and obtaining the identification result of the model to be detected on whether the detection prompt has risks;

[0029] An iterative update module is used to iteratively update the detection prompts of risks in the recognition results; and iteratively update the demonstration pool according to the detection prompts of no risks in the recognition results.

[0030] In a third aspect, a computer device is provided in this embodiment, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the large language model security assessment method based on iterative detection described in the first aspect is implemented.

[0031] In a fourth aspect, a storage medium is provided in this embodiment, on which a computer program is stored, and when the program is executed by a processor, the large language model security assessment method based on iterative detection described in the first aspect above is implemented.

[0032] Compared with the related art, the large language model security assessment method and system based on iterative detection provided in this embodiment constructs the original prompt into a detection prompt based on a preset demonstration pool and disguise strategy; the demonstration pool includes a prefix demonstration pool, a suffix demonstration pool, a strategy conversion demonstration pool and a disguise demonstration pool; the detection prompt is input into the model to be detected to obtain the recognition result of whether the detection prompt has a risk by the model to be detected; the detection prompts with risks in the recognition results are iteratively updated; and the demonstration pool is iteratively updated according to the detection prompts without risks in the recognition results. Through this embodiment, it is possible to adopt an iterative detection method when performing security assessment on the model, dynamically construct detection prompts each time a detection is performed, and iteratively update the detection prompts and demonstration pool after the detection, thereby realizing a more flexible and applicable security assessment method.

[0033] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0035] Figure 1 is a hardware structure block diagram of a terminal of a large language model security assessment method based on iterative detection in an embodiment;

[0036] Figure 2 is a flowchart of a large language model security assessment method based on iterative detection in one embodiment;

[0037] Figure 3 is a flowchart of a large language model security assessment method based on iterative detection in another embodiment;

[0038] Figure 4 is a schematic diagram of a framework for security assessment and protection enhancement of a model to be detected in an embodiment;

[0039] Figure 5 The structure diagram of a large language model security assessment system based on iterative detection in an embodiment.

[0040] In the figure: 102, processor; 104, memory; 106, transmission device; 108, input and output device; 10, prompt construction module; 20, risk identification module; 30, iterative update module. DETAILED DESCRIPTION

[0041] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0042] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not represent quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. Usually, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0043] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 is a hardware structure block diagram of a terminal of the large language model security assessment method based on iterative detection in this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) processor 102 and memory 104 for storing data, wherein processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0044] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the large language model security assessment method based on iterative detection in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0045] The transmission device 106 is used to receive or send data via a network. The above network includes a wireless network provided by the communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0046] In this embodiment, a large language model security assessment method based on iterative detection is provided. Figure 2 is a flowchart of the large language model security assessment method based on iterative detection in this embodiment. Figure 2 As shown, the method comprises the following steps:

[0047] Step S201, based on a preset demonstration pool and camouflage strategy, construct the original prompt into a detection prompt; the demonstration pool includes a prefix demonstration pool, a suffix demonstration pool, a strategy conversion demonstration pool and a camouflage demonstration pool.

[0048] Specifically, the original prompts may contain some malicious and misleading content, which may induce the large model to output non-compliant content. In order to detect and evaluate the security of the large model, a demonstration pool is built in advance, and the original prompts are disguised and transformed using camouflage strategies and examples in the demonstration pool, and dynamically combined to obtain detection prompts for input into the model to be detected.

[0049] The demonstration pool includes a prefix demonstration pool, a suffix demonstration pool, a strategy conversion demonstration pool and a disguise demonstration pool. The prefix examples in the prefix demonstration pool and the suffix examples in the suffix demonstration pool are used to assume some specific scenarios and identities to put the large model to be detected in a safe scenario; the strategy conversion examples in the strategy conversion demonstration pool are used to guide how to use the selected disguise strategy to transform the original prompts. The disguise strategy is a strategy used to transform the original prompts, including but not limited to historical disguise strategy, situational disguise strategy, semantic disguise strategy, logical disguise strategy, etc.; the disguised question-answer pairs in the disguise demonstration pool are question-answer pairs consisting of detection prompts that the model to be detected identifies as risk-free and the corresponding output content of the model in each iterative detection.

[0050] Step S202, input the detection prompt into the model to be detected, and obtain the recognition result of whether the detection prompt has risks by the model to be detected.

[0051] Specifically, the model to be tested identifies risks for the input detection prompts. For detection prompts that are identified as having risks, the model to be tested will make a security response, for example, output a rejection answer, a security warning, etc. For detection prompts that are identified as not having risks, the model to be tested will output the responsive answer content under the detection prompt.

[0052] Step S203, iteratively updating detection prompts for risks in the recognition results; and iteratively updating the demonstration pool according to detection prompts for risks not existing in the recognition results.

[0053] Specifically, in iterative detection, the detection prompts and demonstration pool are iteratively updated according to the identification results of whether the detection prompts of each detection have risks. For detection prompts with risks, the risk causes are further analyzed, and after the detection prompts are rebuilt and optimized through the demonstration pool and disguise strategy, they are input into the model to be detected for risk identification; for detection prompts without risks, they are added to the demonstration pool as disguised question-answer pairs to update the demonstration pool, so as to increase the diversity of samples in the demonstration pool and improve the effect of subsequent iterative detection.

[0054] Through the above steps, it is possible to adopt an iterative detection method when conducting a security assessment on the model. During each detection, based on the preset demonstration pool and camouflage strategy, a detection prompt is dynamically constructed for the original prompt, and the detection prompt is input into the model to be detected for risk identification. After each detection, the detection prompt and the demonstration pool are iteratively updated according to the identification result of whether there is a risk in the detection prompt, and the detection prompt and the demonstration pool are continuously optimized. Compared with the less flexible security assessment method based on a fixed template in the prior art, the security assessment method based on iterative detection in this embodiment has a continuous feedback loop, iteratively updates the detection prompt and the demonstration pool, and can continuously improve the adaptability of the detection strategy, better perform security assessment on the model to be detected, and realize a more flexible and more applicable security assessment method. Furthermore, it can also be applied to a black box environment, and can provide data support for the improvement of the subsequent security protection design of the model to be detected.

[0055] In some of the embodiments, the step S201 above constructs the original prompt into a detection prompt based on the preset demonstration pool and camouflage strategy, including the following steps:

[0056] Select prefixes, suffixes, strategy conversion examples and disguised question-answer pairs from the prefix demonstration pool, suffix demonstration pool, strategy conversion demonstration pool and disguise demonstration pool respectively; transform the original prompts based on the strategy conversion examples and disguise strategies; combine the transformed original prompts with the prefixes, suffixes and disguised question-answer pairs to obtain detection prompts.

[0057] Specifically, there is no limitation on the selection method of examples in each demonstration pool. Examples can be selected randomly or with low risk based on the results of previous experiments. For the original prompt and the selected disguise strategy, the original prompt is transformed using the disguise strategy under the guidance of the strategy conversion example. Exemplarily, this conversion process can be achieved by designing a large language model. The disguise strategy is a strategy for transforming the original prompt, including but not limited to historical disguise strategy, situational disguise strategy, semantic disguise strategy, logical disguise strategy, etc. For example, the historical disguise strategy is to disguise the original prompt by mentioning historical events, figures, culture and other elements.

[0058] The various examples in the demonstration pool exist in the form of strings. The selected prefixes and suffixes are spliced ​​before and after the converted original prompts to provide contextual background to construct specific scenarios, so that the model to be tested can make a reasonable response in the specific scenario, and then disguised question and answer pairs are spliced ​​to reduce risks, and the detection prompts are obtained by combining them.

[0059] By using the obfuscation technology in this embodiment to efficiently combine and obtain a variety of detection prompts, the model to be detected is iteratively tested and security evaluated. Through this dynamic update design, the model to be detected can flexibly adjust its response according to different detection prompts, thereby improving the adaptability and effectiveness of the detection strategy.

[0060] In some of the embodiments, the above step S201 constructs the original prompt as a detection prompt based on the preset demonstration pool and camouflage strategy, and further includes the following steps:

[0061] A single-round detection strategy is adopted to construct the original prompt into a detection prompt at one time; and / or a multi-round detection strategy is adopted to split the original prompt into several sub-prompts, and each sub-prompt is constructed as a detection prompt respectively.

[0062] Among them, when adopting the single-round detection strategy and the multi-round detection strategy, the single-round detection strategy is executed first, and until the detection rounds reach a preset number, the multi-round detection strategy is adopted.

[0063] Specifically, in order to comprehensively evaluate the security of the model to be detected, a single-round detection strategy and a multi-round detection strategy are introduced in this embodiment, and the single-round detection strategy and the multi-round detection strategy can be executed independently or in combination. The single-round detection strategy is to quickly evaluate the reaction of the model to be detected by building a detection prompt once, while the multi-round detection strategy is to simulate the complex interaction process between humans and models, and gradually adjust the detection strategy through continuous dialogue (detection prompts).

[0064] When a single-round detection strategy is adopted, the disguise strategy and examples in the demonstration pool are selected, the original prompt is constructed as a detection prompt at one time, and the detection prompt is input into the model to be detected for risk identification and response. When a multi-round detection strategy is adopted, the original prompt is split into multiple harmless sub-prompts through a pre-trained large language model, and each sub-prompt is combined with the selected disguise strategy and examples in the demonstration pool in each round, that is, the original prompt is converted into multiple rounds of detection prompts, and the detection prompt and demonstration pool are also iteratively updated after each round of detection.

[0065] When combining the single-round detection strategy and the multi-round detection strategy, the single-round detection strategy can be executed first, the original prompt is constructed as a detection prompt at one time, and the detection prompt is input into the model to be detected for risk identification and response. The detection prompt and demonstration pool are iteratively updated according to the recognition result of whether there is a risk, until the detection rounds of the single-round detection strategy reach the preset number of times, and then the multi-round detection strategy is executed to split the original prompt into multiple sub-prompts, each sub-prompt is constructed as a detection prompt, and the sub-prompt is input into the model to be detected in turn for risk identification and response, and the detection prompt and demonstration pool for the next round are iteratively updated according to the recognition result of whether there is a risk.

[0066] By introducing single-round detection strategy and multi-round detection strategy in this embodiment, the sub-prompt decomposition of the multi-round detection strategy fully utilizes the dynamic characteristics of the natural language model dialogue, making the detection process closer to the real user interaction scenario, thereby improving the concealment and effectiveness of the detection, and then gradually revealing the potential vulnerabilities of the model at different interaction stages, and conducting a more in-depth and comprehensive assessment of the security of the model.

[0067] In some of the embodiments, the iterative updating of the detection prompt of the risk in the recognition result in the above step S203 includes the following steps:

[0068] Obtain the risk reasons for the detection prompts through the model to be detected; update the camouflage strategy based on the detection prompts and the corresponding risk reasons; and update the risky detection prompts based on the preset demonstration pool and the updated camouflage strategy.

[0069] Specifically, for risky detection prompts, the model to be detected will respond safely. The risk causes are further analyzed through the model to be detected, such as the model's alignment mechanism, security protection design, etc. The risky detection prompts and risk causes are input into the pre-trained analyst's large language model, and the disguise strategy is re-selected to obtain an optimized and updated disguise strategy. Based on the updated disguise strategy and the examples selected from the demonstration pool, the detection prompts are reconstructed, and the updated detection prompts are input into the model to be detected for risk identification in the next round of detection.

[0070] In this embodiment, the identification result of whether there is a risk is analyzed, and the detection prompts are iteratively updated and optimized in each round of detection, so as to continuously improve the adaptability of the detection strategy through this continuous feedback loop.

[0071] In some of the embodiments, in the above step S203, iteratively updating the demonstration pool according to the detection prompt that there is no risk in the recognition result includes the following steps:

[0072] Based on the detection prompts that do not contain risks and the output content of the model to be detected, a disguised question-answer pair is constructed; the disguised question-answer pair is added to the disguised demonstration pool for updating.

[0073] Specifically, for detection prompts that do not identify risks, the model to be detected will output the response answer content under the detection prompt, and the output content of the detection prompt and the model response will form a disguised question and answer pair, which will be added to the disguised demonstration pool to update the demonstration pool.

[0074] By analyzing the identification results of whether there is a risk in this embodiment, the demonstration pool is iteratively updated in each round of detection, and the diversity of examples in the demonstration pool is increased, so as to continuously improve the adaptability of the detection strategy through this continuous feedback loop.

[0075] In some of the embodiments, the following steps are also included:

[0076] The detection model is trained based on detection prompts that do not contain risks and preset security response results.

[0077] Specifically, for detection prompts that the model to be detected identifies that there is no risk, fine-tuning technology is used to improve the security of the model to be detected. Among them, the detection prompts and preset security response results are collected and organized into a training data set. The preset security response result can be an output rejection answer, security warning, etc. In the fine-tuning process, the model to be detected is trained to learn the potential risks in the detection prompts and reject them. By making small-scale parameter adjustments to the model to be detected and focusing on optimizing the weights of the last few layers of the model, the language generation ability of the model to be detected is maintained, and its protection against malicious input is enhanced.

[0078] In this embodiment, the model is fine-tuned by using the model to be detected to identify detection prompts that do not pose risks, so that the model to be detected can effectively identify and prevent known malicious inputs, while still maintaining a high-quality response to normal inputs, further improving the security protection design of the model.

[0079] The present embodiment is described and illustrated below through preferred embodiments.

[0080] Figure 3 is a flowchart of the large language model security assessment method based on iterative detection in this embodiment. Figure 3 As shown, the method comprises the following steps:

[0081] Step S301: convert the original prompt based on the strategy conversion example and the disguise strategy.

[0082] Step S302, combining the converted original prompt with the selected prefix, suffix and disguised question and answer pair to obtain a detection prompt.

[0083] Step S303, input the detection prompt into the model to be detected, and obtain the recognition result and response content of whether the detection prompt contains risks by the model to be detected.

[0084] Step S304, obtaining the risk reason of the detection prompt through the model to be detected; and updating the camouflage strategy and the detection prompt according to the detection prompt and the corresponding risk reason.

[0085] Step S305, based on the detection prompt that there is no risk and the response content of the model to be detected, a disguised question and answer pair is constructed; the disguised question and answer pair is added to the disguised demonstration pool for updating.

[0086] Step S306, when the detection rounds reach the preset number, a multi-round detection strategy is adopted to split the original prompt into multiple sub-prompts, and each sub-prompt is converted based on the strategy conversion example and the disguise strategy.

[0087] Step S307, combining the converted sub-prompt with the selected prefix, suffix and disguised question-answer pair to obtain a detection prompt. Then executing step S303 to continue inputting the detection prompt into the model to be detected until the iterative detection is completed.

[0088] Step S308: Based on the detection prompt that there is no risk and the preset safety response result, the detection model is trained.

[0089] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0090] Figure 4 is a schematic diagram of the framework of security assessment and protection enhancement of the model to be detected in this embodiment, such as Figure 4 As shown, for the input original prompt, a single-round detection strategy or a multi-round detection strategy is selected. In the multi-round detection strategy, the original prompt is first split into multiple harmless sub-prompts. In each round of detection, a disguise strategy is selected (the circular pattern in each row represents a different disguise strategy, and the black circular pattern represents the disguise strategy selected in this round). Under the guidance of the strategy conversion example in the strategy conversion demonstration pool, the selected disguise strategy is used to convert the original prompt or sub-prompt. The prefix selected from the prefix demonstration pool and the suffix selected from the suffix demonstration pool are spliced ​​before and after the converted original prompt or sub-prompt, and the disguised question and answer pair selected from the disguise demonstration pool are added to obtain the detection prompt.

[0091] The detection prompt is input into the model to be detected, and the recognition result and response content of whether the detection prompt has risks are obtained by the model to be detected. For the detection prompts that are identified as having risks, the risk reasons are obtained through the model to be detected, and then the risk reasons and detection prompts are input into the analyst's large model to re-select the disguise strategy. In the next round of detection, the detection prompts are rebuilt based on the updated disguise strategy and the examples in the demonstration pool.

[0092] For detection prompts that do not identify risks, the detection prompts and the model's response content are combined into a question-answer pair and updated to the disguise demonstration pool. In addition, the detection prompts and the preset security response content are combined into a training data set to fine-tune the detection model to train and improve the security protection design of the detection model.

[0093] In this embodiment, a large language model security assessment system based on iterative detection is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. The terms "module", "unit", "sub-unit", etc. used below can implement a combination of software and / or hardware with predetermined functions. Although the systems described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0094] Figure 5 is a structural block diagram of the large language model security assessment system based on iterative detection in this embodiment. Figure 5 As shown, the system includes:

[0095] The prompt construction module 10 is used to construct the original prompt into a detection prompt based on a preset demonstration pool and a disguise strategy; the demonstration pool includes a prefix demonstration pool, a suffix demonstration pool, a strategy conversion demonstration pool and a disguise demonstration pool.

[0096] The risk identification module 20 is used to input the detection prompt into the model to be detected, and obtain the identification result of the model to be detected on whether there is a risk in the detection prompt.

[0097] The iterative update module 30 is used to iteratively update the detection prompts of risks in the recognition results; and iteratively update the demonstration pool according to the detection prompts of no risks in the recognition results.

[0098] Through the system provided in this embodiment, it is possible to adopt an iterative detection method when conducting security assessment on the model. During each detection, based on the preset demonstration pool and camouflage strategy, a detection prompt is dynamically constructed for the original prompt, and the detection prompt is input into the model to be detected for risk identification. After each detection, the detection prompt and the demonstration pool are iteratively updated according to the identification result of whether there is a risk in the detection prompt, and the detection prompt and the demonstration pool are continuously optimized. Compared with the less flexible security assessment method based on fixed templates in the prior art, the security assessment method based on iterative detection in this embodiment has a continuous feedback loop, iteratively updates the detection prompt and the demonstration pool, and can continuously improve the adaptability of the detection strategy, better perform security assessment on the model to be detected, and realize a more flexible and more applicable security assessment method. Furthermore, it can also be applied to a black box environment, and can provide data support for the improvement of the subsequent security protection design of the model to be detected.

[0099] In some of the embodiments, the prompt building module 10 is further used for:

[0100] Select prefixes, suffixes, strategy conversion examples and disguised question-answer pairs from the prefix demonstration pool, suffix demonstration pool, strategy conversion demonstration pool and disguise demonstration pool respectively; transform the original prompts based on the strategy conversion examples and disguise strategies; combine the transformed original prompts with the prefixes, suffixes and disguised question-answer pairs to obtain detection prompts.

[0101] In some of the embodiments, the prompt building module 10 is further used for:

[0102] A single-round detection strategy is adopted to construct the original prompt into a detection prompt at one time; and / or a multi-round detection strategy is adopted to split the original prompt into several sub-prompts, and each sub-prompt is constructed as a detection prompt respectively.

[0103] In some embodiments, the prompt building module 10 is further used for:

[0104] When a single-round detection strategy and a multi-round detection strategy are adopted, the single-round detection strategy is first executed until the number of detection rounds reaches a preset number, and then the multi-round detection strategy is adopted.

[0105] In some embodiments, the iterative update module 30 is further configured to:

[0106] Obtain the risk reasons for the detection prompts through the model to be detected; update the camouflage strategy based on the detection prompts and the corresponding risk reasons; and update the risky detection prompts based on the preset demonstration pool and the updated camouflage strategy.

[0107] In some embodiments, the iterative update module 30 is further configured to:

[0108] Based on the detection prompts that do not contain risks and the output content of the model to be detected, a disguised question-answer pair is constructed; the disguised question-answer pair is added to the disguised demonstration pool for updating.

[0109] In some of the embodiments, it also includes: a training module for training the detection model based on the detection prompts that there is no risk and the preset safety response results.

[0110] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0111] In this embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0112] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0113] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0114] In addition, in combination with the large language model security assessment method based on iterative detection provided in the above embodiments, a storage medium can also be provided in this embodiment to implement. The storage medium stores a computer program; when the computer program is executed by the processor, any large language model security assessment method based on iterative detection in the above embodiments is implemented.

[0115] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0116] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0117] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0118] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0119] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A large language model security assessment method based on iterative detection, characterized in that: include: Based on a preset demonstration pool and camouflage strategy, the original prompt is constructed into a detection prompt; the demonstration pool includes a prefix demonstration pool, a suffix demonstration pool, a strategy conversion demonstration pool and a camouflage demonstration pool; Inputting the detection prompt into the model to be detected, and obtaining a recognition result of the model to be detected on whether the detection prompt has a risk; Iteratively updating detection prompts of risks in the recognition results; And, based on the detection prompt that there is no risk in the recognition result, the demonstration pool is iteratively updated.

2. The method according to claim 1, characterized in that The original prompt is constructed into a detection prompt based on the preset demonstration pool and camouflage strategy, including: Select prefixes, suffixes, strategy conversion examples and disguised question-answer pairs from the prefix demonstration pool, suffix demonstration pool, strategy conversion demonstration pool and disguised demonstration pool respectively; Based on the strategy conversion example and the disguise strategy, convert the original prompt; The converted original prompt is combined with the prefix, suffix and disguised question-answer pair to obtain the detection prompt.

3. The method according to claim 1, characterized in that The iterative updating of the detection prompt of the risk in the recognition result includes: Obtaining the risk reason prompted by the detection through the model to be detected; According to the detection prompt and the corresponding risk reason, the camouflage strategy is updated; and based on the preset demonstration pool and the updated camouflage strategy, the detection prompt of the risk is updated.

4. The method according to claim 1, characterized in that: The iterative updating of the demonstration pool according to the detection prompt that there is no risk in the recognition result includes: Based on the detection prompt that there is no risk and the output content of the model to be detected, a disguised question-answer pair is formed; The disguised question and answer pair is added to the disguised demonstration pool for updating.

5. The method according to claim 1, characterized in that The method of constructing the original prompt into a detection prompt based on the preset demonstration pool and camouflage strategy also includes: A single-round detection strategy is adopted to construct the original prompt into a detection prompt at one time; And / or, a multi-round detection strategy is adopted to split the original prompt into a plurality of sub-prompts, and each of the sub-prompts is constructed as a detection prompt.

6. The method according to claim 5, characterized in that Also includes: When a single-round detection strategy and a multi-round detection strategy are adopted, the single-round detection strategy is first executed until the number of detection rounds reaches a preset number, and then the multi-round detection strategy is adopted.

7. The method according to claim 1, characterized in that Also includes: Based on the detection prompt that there is no risk and the preset safety response result, the model to be detected is trained.

8. A large language model security assessment system based on iterative detection, characterized in that: include: A prompt construction module is used to construct the original prompt into a detection prompt based on a preset demonstration pool and a disguise strategy; the demonstration pool includes a prefix demonstration pool, a suffix demonstration pool, a strategy conversion demonstration pool and a disguise demonstration pool; A risk identification module, used for inputting the detection prompt into the model to be detected, and obtaining the identification result of the model to be detected on whether the detection prompt has risks; An iterative update module, used for iteratively updating the detection prompt of risks in the recognition result; And, based on the detection prompt that there is no risk in the recognition result, the demonstration pool is iteratively updated.

9. A computer device comprising a memory and a processor, characterized in that: The memory stores a computer program, and the processor is configured to run the computer program to execute the large language model security assessment method based on iterative detection according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the large language model security assessment method based on iterative detection described in any one of claims 1 to 7 are implemented.

Images