A model attack method, device, storage medium and equipment

Abstract

The present application provides a model attack method, device, storage medium and equipment, which is applied in the field of computer technology. The method includes: the attack model sends a first attack prompt related to the attack task to the target model, performs semantic analysis on the first response content returned by the target model for the first attack prompt, determines whether to stop the attack according to the semantic analysis result, and if the attack continues, determines the dialogue direction with the target model according to the attack prompt, the response content and the attack correlation score between the attack task. In the process of attacking the target model, through multiple rounds of dialogue, the response content is semantically analyzed, the dialogue direction is gradually adjusted, and the attack success rate is improved, so that researchers can identify the weak links of the target model when generating the response content, which helps to discover the vulnerabilities of the target model and develop more effective security defense mechanisms.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to a model attack method, apparatus, storage medium and device in the field of computer technology. Background Art

[0002] Large Language Models (LLMs) are trained using large amounts of text data and complex feedback mechanisms, and are able to demonstrate superior processing capabilities in Natural Language Processing (NLP) tasks. However, a series of attacks known as "jailbreaks" exploit LLM vulnerabilities to undermine LLM's security mechanisms and induce LLM to output responses that are inconsistent with expectations. Research on jailbreak attacks helps reveal LLM's systemic vulnerabilities and promotes the improvement of LLM's security defense capabilities. Summary of the invention

[0003] The present application provides a model attack method, device, storage medium and equipment, which can perform semantic analysis on the response content returned by the target model in response to the attack prompt, and timely adjust the subsequent dialogue direction, or determine the completion of the attack task. In the process of attacking the target model, through multiple rounds of dialogue, the response content is semantically analyzed, the dialogue direction is gradually adjusted, and the attack success rate is improved, so that researchers can identify the weak links of the target model when generating the response content, which helps to discover the vulnerabilities of the target model and develop more effective security defense mechanisms.

[0004] In a first aspect, a model attack method is provided, the method comprising: sending a first attack prompt related to an attack task to a target model in a first dialogue round; receiving a first response content returned by the target model for the first attack prompt; performing semantic analysis on the first response content to determine whether the first response content conforms to a target response content corresponding to the attack task; if the first response content does not conform to the target response content, determining a dialogue direction with the target model based on the first response content, the first attack prompt, and an attack correlation score between the attack task; if the first response content conforms to the target response content, determining that the attack task is completed.

[0005] Through the above technical solution, the attack model sends the first attack prompt related to the attack task to the target model, and performs semantic analysis on the first response content returned by the target model to the first attack prompt. If the first response content meets the target response content corresponding to the attack task, the attack task is directly determined to be completed; if the first response content does not meet the target response content, the direction of the dialogue with the target model is determined based on the attack correlation score between the first response content and the first attack prompt and the attack task. During the attack process against the target model, through multiple rounds of dialogue, the response content is semantically analyzed, the dialogue direction is gradually adjusted, and the attack success rate is improved, so that researchers can identify the weak links of the target model when generating the response content, which helps to discover the vulnerabilities of the target model and develop more effective security defense mechanisms.

[0006] In combination with the first aspect, in some possible implementations, the step of determining the dialogue direction with the target model based on the attack relevance score between the first response content, the first attack prompt, and the attack task if the first response content does not conform to the target response content includes: if the first response content does not conform to the target response content, determining the attack relevance score based on the relevance of the first response content to the first attack prompt and the attack task; comparing the attack relevance score with the historical relevance score corresponding to the second dialogue round, the second dialogue round being the previous dialogue round of the first dialogue round; if the attack relevance score is less than or equal to the historical relevance score, determining the dialogue direction with the target model to restate the first dialogue round or to restate the second dialogue round; if the attack relevance score is greater than the historical relevance score, determining the dialogue direction with the target model to enter a third dialogue round, the third dialogue round being the next dialogue round of the first dialogue round.

[0007] In combination with the first aspect and the above-mentioned implementation methods, in some possible implementation methods, if the attack relevance score is less than or equal to the historical relevance score, the step of determining the direction of the dialogue with the target model to restate the first dialogue round or to restate the second dialogue round includes: if the attack relevance score is less than or equal to the historical relevance score, comparing the number of repetitions of the first attack prompt in the first dialogue round with a preset repetition number threshold; if the number of repetitions is less than the preset repetition number threshold, determining the direction of the dialogue with the target model to restate the first dialogue round; if the number of repetitions is equal to the preset repetition number threshold, determining the direction of the dialogue with the target model to restate the second dialogue round.

[0008] In combination with the first aspect and the above-mentioned implementation methods, in some possible implementation methods, if the number of repetitions is less than a preset repetition number threshold, determining the direction of the dialogue with the target model as a step of retelling the first dialogue round includes: if the number of repetitions is less than the preset repetition number threshold, optimizing the first attack prompt using an optimization strategy, the optimization strategy including at least one of vocabulary replacement, detail addition and modification, semantic expansion, context association, structure adjustment and complexity adjustment; sending the optimized first attack prompt to the target model.

[0009] In combination with the first aspect and the above-mentioned implementation methods, in some possible implementation methods, if the number of repetitions is equal to a preset repetition number threshold, determining the direction of the dialogue with the target model is a step of retelling the second dialogue round, including: if the number of repetitions is equal to the preset repetition number threshold, withdrawing the first dialogue round; adjusting the second attack prompt corresponding to the second dialogue round based on the prompt angle; and sending the adjusted second attack prompt to the target model.

[0010] In combination with the first aspect and the above-mentioned implementation methods, in some possible implementation methods, after the step of determining that the dialogue direction with the target model is to enter the third dialogue round if the attack relevance score is greater than the historical relevance score, the method further includes: in response to the dialogue direction entering the third dialogue round, the number of dialogue rounds of the first dialogue round in the dialogue is incremented to obtain the number of dialogue rounds of the third dialogue round in the dialogue; the number of dialogue rounds of the third dialogue round in the dialogue is determined as the current dialogue round, and the current dialogue round is compared with a preset round threshold; if the current dialogue round is less than or equal to the preset round threshold, a third attack prompt corresponding to the third dialogue round is generated based on the first dialogue round and the second dialogue round, and the third attack prompt is sent to the target model; if the current dialogue round is greater than the preset round threshold, it is determined that the attack task is completed.

[0011] In combination with the first aspect and the above-mentioned implementation methods, in some possible implementation methods, before the step of sending the first attack prompt related to the attack task to the target model in the first dialogue round, it also includes: receiving the attack task, obtaining a non-attack topic related to the attack task; and constructing a basic dialogue for the attack task based on the non-attack topic.

[0012] In combination with the first aspect and the above-mentioned implementation methods, in some possible implementation methods, the step of sending a first attack prompt related to the attack task to the target model in the first dialogue round includes: obtaining attack words related to the attack task; introducing the attack words into the basic dialogue to obtain the first attack prompt related to the attack task, and sending the first attack prompt to the target model in the first dialogue round.

[0013] In a second aspect, a model attack device is provided, the device comprising:

[0014] A first attack unit, used to send a first attack prompt related to an attack task to a target model in a first dialogue round;

[0015] A response receiving unit, configured to receive a first response content returned by the target model to the first attack prompt;

[0016] A semantic analysis unit, configured to perform semantic analysis on the first response content, determine the relevance of the first response content with the first attack prompt and the attack task, and obtain an attack relevance score;

[0017] A dialogue direction determination unit, configured to determine a dialogue direction with a target model based on the number of dialogue turns and the attack relevance score if the attack relevance score does not reach a preset score threshold;

[0018] The attack task completion unit is used to determine that the attack task is completed if the attack relevance score reaches a preset score threshold.

[0019] In a third aspect, a computer device is provided, including a processor, a memory, and an input and output interface;

[0020] The processor is connected to the memory and the input and output interface respectively, wherein the input and output interface is used for page interaction, the memory is used for storing program codes, and the processor is used for calling the program codes to execute the above method steps.

[0021] In a fourth aspect, a computer storage medium is provided, wherein the computer storage medium stores a computer program, and the computer program includes program instructions. When the program instructions are executed by a processor, the above-mentioned method steps are executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a system architecture diagram of a model attack method provided in an embodiment of the present application;

[0023] Figure 2 It is a flowchart of a model attack method provided in an embodiment of the present application;

[0024] Figure 3 It is a flowchart of a model attack method provided in an embodiment of the present application;

[0025] Figure 4 It is a structural schematic diagram of a model attack device provided in an embodiment of the present application;

[0026] Figure 5 It is a structural schematic diagram of a model attack device provided in an embodiment of the present application;

[0027] Figure 6 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The technical solution in the present application will be described clearly and in detail below in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B: "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0029] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.

[0030] See also Figure 1 , Figure 1 : is a system architecture diagram of a model attack method provided by an embodiment of the present application. Figure 1 As shown, the model attack method provided in the embodiment of the present application can be applied to the attack model, and the attack model is used to implement a jailbreak attack on the target model. The model attack device provided in the embodiment of the present application can be a terminal device, including but not limited to: user terminals such as personal computers, tablet computers, handheld devices, vehicle-mounted devices or wearable devices. Through the embodiment of the present application, in the process of attacking the target model, through multiple rounds of dialogue, the response content is semantically analyzed, the dialogue direction is gradually adjusted, and the attack success rate is improved, so that researchers can identify the weak links of the target model when generating the response content, which is helpful to discover the vulnerabilities of the target model and develop a more effective security defense mechanism.

[0031] based on Figure 1 The system architecture diagram shown below will be combined with Figure 2-Figure 3 , the model attack method provided in the embodiment of the present application is introduced in detail.

[0032] See also Figure 2 , Figure 2 Schematic diagram of a model attack method provided in an embodiment of the present application. Figure 2 As shown, the method of the embodiment of the present application may include the following steps S101-S105.

[0033] S101, sending a first attack prompt related to an attack task to a target model in a first dialogue round;

[0034] Specifically, the model attack method provided in the embodiment of the present application is a multi-round attack, that is, to lure the target model to attack through multiple rounds of dialogue. In the first round of dialogue, the first attack prompt related to the attack task is sent to the target model. The target model is LLM. The purpose of the attack task is to guide the target model to generate content that violates platform guidelines or social ethics. The attack prompt is the input prompt used by the attack model to guide the target model. Researchers can use the model attack method to reversely infer the weak links of the target model when facing attacks, thereby improving the security mechanism of the target model.

[0035] S102, receiving a first response content returned by the target model to the first attack prompt;

[0036] Specifically, each dialogue round includes an attack prompt sent by the attack model to the target model, and a response content returned by the target model. Therefore, when the attack model sends the first attack prompt to the target model, the target model responds to the attack prompt, and the attack model receives the first response content returned by the target model to the first attack prompt.

[0037] S103, performing semantic analysis on the first response content to determine whether the first response content conforms to the target response content corresponding to the attack task;

[0038] Specifically, the first response content is the reply made by the target model to the first attack prompt. The semantic analysis technology is used to perform semantic analysis on the first response content to obtain the similarity between the first response content and the target response content corresponding to the attack task. The similarity is compared with a preset similarity threshold. According to the comparison result, it is determined whether the first response content conforms to the target response content corresponding to the attack task. The target response content is the harmful response content corresponding to the attack task expected by the attack model.

[0039] S104, if the first response content does not match the target response content, determining the dialogue direction with the target model based on the first response content, the first attack prompt, and the attack relevance score between the attack tasks;

[0040] Specifically, if the similarity is less than the preset similarity threshold, it is determined that the first response content does not conform to the target response content, that is, the attack task has not been completed. Based on the first response content, the first attack prompt, and the attack relevance score between the attack tasks, the conversation direction with the target model is determined. The attack relevance score is used to determine the semantic gain of each round of conversation. The conversation direction includes forward and restatement. Forward means entering the next round of conversation, and restatement includes restating the current round of conversation or the previous round of conversation. The ultimate goal of the model attack method is to induce the target model to generate harmful response content related to the attack task. In order to achieve this goal, the semantic relevance between the attack prompt and its corresponding response content and the attack task should be gradually enhanced in multiple rounds of conversation, that is, the attack relevance score should continue to grow until the attack task is completed.

[0041] S105: If the first response content matches the target response content, it is determined that the attack task is completed.

[0042] Specifically, if the similarity is greater than or equal to the preset similarity threshold, it is determined that the first response content is consistent with the target response content, that is, the target model has output the harmful response content corresponding to the attack task expected by the attack model, and it is determined that the attack task is completed and the attack on the target model is stopped. Researchers can study the multi-round dialogue between the attack model and the target model during the attack process to discover the vulnerabilities of the target model in processing multi-round dialogues, and then promote the defense capability of the target model's dialogue system when processing multi-round dialogues.

[0043] In an embodiment of the present application, the attack model sends a first attack prompt related to the attack task to the target model, and performs semantic analysis on the first response content returned by the target model for the first attack prompt. If the first response content meets the target response content corresponding to the attack task, the attack task is directly determined to be completed; if the first response content does not meet the target response content, the direction of the dialogue with the target model is determined based on the attack correlation score between the first response content and the first attack prompt and the attack task. During the attack process against the target model, through multiple rounds of dialogue, the response content is semantically analyzed, the dialogue direction is gradually adjusted, and the attack success rate is improved, so that researchers can identify the weak links of the target model when generating the response content, which helps to discover the vulnerabilities of the target model and develop more effective security defense mechanisms.

[0044] See also Figure 3 , Figure 3 Schematic diagram of a model attack method provided in an embodiment of the present application. Figure 3 As shown, the method of the embodiment of the present application may include the following steps S201-S210.

[0045] S201, receiving an attack task, obtaining a non-attack topic related to the attack task, and building a basic dialogue for the attack task based on the non-attack topic;

[0046] Specifically, after receiving the attack task, the attack model obtains non-attack topics related to the attack task from its own corpus, starts with the non-attack topics, builds a basic dialogue for the attack task based on the non-attack topics, and conducts a dialogue with the target model based on the basic dialogue to build the target model's initial understanding of the attack task. The attack model is a pre-trained model with natural language processing capabilities that can understand grammar, semantics, and context. The corpus is a systematic and computerized collection of real languages ​​used for language analysis and corpus analysis in the attack model, that is, a database for storing language materials. Non-attack topics are topics related to the attack task but harmless that the attack model obtains from the corpus. For example, occupations, daily activities, application scenarios, etc. related to the attack task are not easily detected by the target model, but provide support for the subsequent guidance of the target model.

[0047] S202, acquiring attack words related to the attack task, introducing the attack words into the basic dialogue, obtaining a first attack prompt related to the attack task, and sending the first attack prompt to the target model in a first dialogue round;

[0048] Specifically, the model attack method provided in the embodiment of the present application is a multi-round attack, that is, to lure and attack the target model through multiple rounds of dialogue. The attack model starts a dialogue with the target model through a non-attack topic, and at the same time obtains the attack words related to the attack task, introduces the attack words into the basic dialogue according to the context information of the dialogue with the target model, obtains the first attack prompt related to the attack task in the first dialogue round, and sends the first attack prompt to the target model in the first dialogue round, and the first dialogue round is the current dialogue round. Among them, the target model is LLM, the attack words are key phrases or concepts related to the attack task, the purpose of the attack task is to guide the target model to generate content that violates platform guidelines or social ethics, and the attack prompt is the input prompt used by the attack model to guide the target model. Researchers can use the model attack method to reverse the weak links of the target model when facing attacks, thereby improving the security mechanism of the target model. In the multi-round dialogue with the target model, the attack model continuously analyzes the dialogue context and its semantic features, and adjusts the subsequent attack prompts according to the content of each response of the model through a carefully designed feedback mechanism, so that it is closer to the dialogue context and attack intention.

[0049] S203, receiving a first response content returned by the target model to the first attack prompt;

[0050] Specifically, each dialogue round includes an attack prompt sent by the attack model to the target model, and a response content returned by the target model. Therefore, when the attack model sends the first attack prompt to the target model, the target model responds to the attack prompt, and the attack model receives the first response content returned by the target model to the first attack prompt.

[0051] S204, performing semantic analysis on the first response content to determine whether the first response content conforms to the target response content corresponding to the attack task;

[0052] Specifically, the first response content is the reply made by the target model to the first attack prompt. The semantic analysis technology is used to perform semantic analysis on the first response content to obtain the similarity between the first response content and the target response content corresponding to the attack task. The similarity is compared with a preset similarity threshold. According to the comparison result, it is determined whether the first response content conforms to the target response content corresponding to the attack task. The target response content is the harmful response content corresponding to the attack task expected by the attack model.

[0053] Optionally, an independent semantic analysis model can be used for semantic analysis, or a semantic analysis model can be set in the attack model for semantic analysis. The semantic analysis model is an LLM defined using a specific prompt, and the prompt word should directly and specifically describe the target response content of the attack task. Before performing semantic analysis, the first response content and the target response content are preprocessed to reduce text noise and improve the accuracy of semantic analysis. The preprocessing includes but is not limited to word segmentation, part-of-speech tagging, special character processing, etc. The semantic analysis method can be to convert the preprocessed text content into a word vector sequence, and calculate the similarity between texts by methods such as cosine similarity or Euclidean distance.

[0054] S205, if the first response content does not match the target response content, determining an attack relevance score based on the relevance of the first response content, the first attack prompt, and the attack task;

[0055] Specifically, if the similarity is less than a preset similarity threshold, it is determined that the first response content does not conform to the target response content, that is, the attack task has not been completed. Based on the correlation between the first response content, the first attack prompt, and the attack task, the attack relevance score is determined, and the attack relevance score is used to determine the semantic gain of each round of dialogue. The process of determining the attack relevance score can also be performed in the semantic analysis model provided in step S204. The ultimate goal of the model attack method is to induce the target model to generate harmful response content related to the attack task. In order to achieve this goal, the semantic relevance between the attack prompt and its corresponding response content and the attack task should be gradually enhanced in multiple rounds of dialogue, that is, the attack relevance score should continue to grow until the attack task is completed.

[0056] S206, comparing the attack relevance score with the historical relevance score corresponding to the second dialogue round;

[0057] Specifically, in order to determine whether the attack relevance score continues to grow, the attack relevance score is compared with the historical relevance score corresponding to the second dialogue round, and the dialogue direction with the target model is determined based on the comparison result. Among them, the second dialogue round is the previous dialogue round of the first dialogue round, and the historical relevance score is the relevance score corresponding to the second dialogue round. The dialogue direction includes forward and restatement. Forward means entering the next dialogue round, and restatement includes restating the current dialogue round or the previous dialogue round. It should be noted that if the first dialogue round is the first dialogue round in multiple dialogue rounds, the historical relevance score is determined to be zero, and the dialogue direction with the target model is determined to enter the next dialogue round.

[0058] S207, if the attack relevance score is less than or equal to the history relevance score, determining the conversation direction with the target model to be restatement of the first conversation turn or restatement of the second conversation turn;

[0059] Specifically, if the attack relevance score is less than or equal to the historical relevance score, it is considered that the semantic relevance between the attack prompt and the response content and the attack task has not been enhanced, and the dialogue direction with the target model is determined to be a restatement of the first dialogue round or a restatement of the second dialogue round. Since the number of restatements is limited, the number of restatements of the first attack prompt in the first dialogue round is compared with a preset restatement number threshold.

[0060] If the number of repetitions is less than the preset repetition threshold, the dialogue direction with the target model is determined to be a repetition of the first dialogue round. In this case, it can be considered that the semantic relevance between the first attack prompt and its first response content in the first dialogue round and the attack task has not been improved, but the dialogue content has not obviously deviated from the attack path. The first attack prompt in the first dialogue round is re-stated. The specific process is to optimize the first attack prompt. The specific process of optimization is to optimize the first attack prompt using an optimization strategy. The optimization strategy includes at least one of vocabulary replacement, detail addition and modification, semantic expansion, context association, structural adjustment and complexity adjustment, and send the optimized first attack prompt to the target model. Among them, vocabulary replacement is to use more accurate or obscure vocabulary to replace the words in the attack prompt before optimization to enhance the contextual semantic relevance. Detail addition and modification is to add or modify the details in the prompt to make it more directly related to the attack task, for example, by proposing more detailed steps or adding situational descriptions, to guide the target model to generate response content that is closer to the attack task. Semantic expansion is to enrich the expression of attack prompts with synonyms or related concepts to make them more risky, but at the same time keep the maliciousness low to reduce the risk of being detected by security mechanisms. For example, general chemical reactions are expanded into specific explosive material production processes to imply the association with attack tasks. Context association is to strengthen the connectivity between attack prompts and the content of the dialogue in historical dialogue rounds, ensuring that the attack prompts in each dialogue round are closely connected with the previous information flow, thereby gradually guiding the target model to form an overall understanding of the attack task. Structural adjustment is to rearrange or rephrase the sentences in the attack prompts to make their semantic structure closer to the attack task, for example, changing the way the question is asked or adding relevant background information. Complexity adjustment is to adjust the semantic depth and complexity of the attack prompts so that the target model generates more specific and harmful content, for example, from simple descriptions to complex operation steps.

[0061] If the number of repetitions is equal to the preset repetition threshold, the direction of the dialogue with the target model is determined to be to retell the second dialogue round. In this case, it can be considered that the first attack prompt of the first dialogue round causes the response content of the target model to deviate seriously from expectations, and it cannot be corrected by optimizing the first attack prompt. It is necessary to adjust the second attack prompt corresponding to the second dialogue round and send the adjusted second attack prompt to the target model. The adjustment is to regenerate the second attack prompt to redirect the target model from a new angle. The specific process of the adjustment is to withdraw the first dialogue round and adjust the second attack prompt corresponding to the second dialogue round based on the prompt angle.

[0062] It should be noted that the number of conversation rounds of the first conversation round in the multi-round conversation is determined as the current conversation round number. If the first conversation round is repeated, the current conversation round number remains unchanged; if the second conversation round is repeated, the current conversation round number is decremented, and the conversation round number after the decrement operation is determined as the current conversation round number. For example, if the number of conversation rounds of the first conversation round in the multi-round conversation is 3, if the first conversation round is repeated, the current conversation round number is 3; if the second conversation round is repeated, the current conversation round number is 2.

[0063] S208, if the attack relevance score is greater than the history relevance score, determining the direction of the dialogue with the target model to enter the third dialogue round;

[0064] Specifically, if the attack relevance score is greater than the historical relevance score, it is considered that the semantic relevance between the attack prompt and the response content and the attack task is enhanced, and the dialogue direction between the attack model and the target model is determined to be forward, that is, entering the third dialogue round, which is the next dialogue round of the first dialogue round. The context of the first dialogue round is used to design the attack prompt of the third dialogue round, so that while maintaining contextual coherence, it can further guide the target model to approach the attack task.

[0065] In a feasible implementation, in order to prevent triggering the security defense mechanism of the target model, a preset round number threshold is pre-set. If the number of dialogue rounds reaches the preset round number threshold, it is determined that the attack task is completed, and the attack model is controlled to stop attacking the target model. Therefore, after determining that the dialogue direction is to enter the third dialogue round, in response to the dialogue direction of entering the third dialogue round, the number of dialogue rounds of the first dialogue round in the dialogue is incremented to obtain the number of dialogue rounds of the third dialogue round in the dialogue, and the number of dialogue rounds of the third dialogue round in the dialogue is determined as the current dialogue round number, and the current dialogue round number is compared with the preset round number threshold. If the current dialogue round number is less than or equal to the preset round number threshold, a third attack prompt corresponding to the third dialogue round is generated based on the first dialogue round and the second dialogue round, and the third attack prompt is sent to the target model; if the current dialogue round number is greater than the preset round number threshold, it is determined that the attack task is completed. For example, if the number of dialogue rounds of the first dialogue round in multiple dialogue rounds is 3, and the preset dialogue round threshold is 3, if the third dialogue round is entered, the number of dialogue rounds of the first dialogue round is incremented, and the current number of dialogue rounds is 4, and the attack on the target model is stopped.

[0066] S209: If the first response content matches the target response content, it is determined that the attack task is completed.

[0067] Specifically, if the similarity is greater than or equal to the preset similarity threshold, it is determined that the first response content is consistent with the target response content, that is, the target model has output the harmful response content corresponding to the attack task expected by the attack model, and it is determined that the attack task is completed and the attack on the target model is stopped. Researchers can study the multi-round dialogue between the attack model and the target model during the attack process to discover the vulnerabilities of the target model in processing multi-round dialogues, and then promote the defense capability of the target model's dialogue system when processing multi-round dialogues.

[0068] In the embodiment of the present application, after the attack model receives the attack task, it obtains the non-attack topic related to the attack task, builds the basic dialogue for the attack task based on the non-attack topic, obtains the attack word related to the attack task after the target model has established the initial cognition of the attack task, introduces the attack word into the basic dialogue, obtains the first attack prompt related to the attack task, and sends the first attack prompt to the target model in the first dialogue round to receive the first response content returned by the target model for the first attack prompt, and then performs semantic analysis on the first response content, and judges whether the first response content conforms to the target response content corresponding to the attack task according to the result of the semantic analysis, and then determines whether to stop the attack. If the attack continues, the dialogue direction with the target model is determined according to the attack relevance score between the attack prompt, the response content and the attack task, so that the response content output by the target model gradually approaches the target response content corresponding to the attack task. Through multi-round dialogue, the response content is semantically analyzed, the dialogue direction is gradually adjusted, and the attack success rate is improved, so that researchers can identify the weak links of the target model when generating the response content, which helps to discover the vulnerabilities of the target model and develop more effective security defense mechanisms. This method can also help researchers discover consistency issues of the target model in multi-round dialogue interactions, as well as possible loopholes when dealing with complex contexts, thereby promoting dialogue systems to more effectively handle various challenges of multi-round dialogues during design, training, and deployment.

[0069] based on Figure 1 The system architecture diagram will be combined with Figure 4 , the model attack device provided in the embodiment of the present application is introduced in detail. It should be noted that, Figure 4 The model attack device in is used to execute this application Figure 2-Figure 3 For the convenience of explanation, only the part related to the embodiment of the present application is shown. For the specific technical details not disclosed, please refer to the present application. Figure 2-Figure 3 The embodiment shown.

[0070] See also Figure 4 , Figure 4 Schematic diagram of a model attack device provided in an embodiment of the present application. Figure 4As shown, the model attack device 1 of the embodiment of the present application may include: a first attack unit 11, a response receiving unit 12, a semantic analysis unit 13, a dialogue direction determination unit 14 and an attack task completion unit 15.

[0071] A first attack unit 11, configured to send a first attack prompt related to an attack task to a target model in a first dialogue round;

[0072] A response receiving unit 12, configured to receive a first response content returned by the target model to the first attack prompt;

[0073] The semantic analysis unit 13 is used to perform semantic analysis on the first response content to determine whether the first response content conforms to the target response content corresponding to the attack task;

[0074] A dialogue direction determining unit 14, configured to determine a dialogue direction with a target model based on the first response content, the first attack prompt, and an attack correlation score between the attack tasks if the first response content does not match the target response content;

[0075] The attack task completion unit 15 is used to determine that the attack task is completed if the first response content matches the target response content.

[0076] In an embodiment of the present application, the attack model sends a first attack prompt related to the attack task to the target model, and performs semantic analysis on the first response content returned by the target model for the first attack prompt. If the first response content meets the target response content corresponding to the attack task, the attack task is directly determined to be completed; if the first response content does not meet the target response content, the direction of the dialogue with the target model is determined based on the attack correlation score between the first response content and the first attack prompt and the attack task. During the attack process against the target model, through multiple rounds of dialogue, the response content is semantically analyzed, the dialogue direction is gradually adjusted, and the attack success rate is improved, so that researchers can identify the weak links of the target model when generating the response content, which helps to discover the vulnerabilities of the target model and develop more effective security defense mechanisms.

[0077] See also Figure 5 , Figure 5 Schematic diagram of a model attack device provided in an embodiment of the present application. Figure 5 As shown, the model attack device 1 of the embodiment of the present application may include: a non-attack topic acquisition unit 16, a basic dialogue construction unit 17, a first attack unit 11, a response receiving unit 12, a semantic analysis unit 13, a dialogue direction determination unit 14 and an attack task completion unit 15.

[0078] A non-attack topic acquisition unit 16, configured to receive an attack task and acquire a non-attack topic related to the attack task;

[0079] A basic dialogue construction unit 17, used to construct a basic dialogue for an attack task based on a non-attack topic;

[0080] A first attack unit 11, configured to send a first attack prompt related to an attack task to a target model in a first dialogue round;

[0081] Optionally, the first attack unit 11 may include:

[0082] An attack word acquisition subunit 111, used to acquire attack words related to the attack task;

[0083] The attack prompt generation subunit 112 is used to introduce the attack word into the basic dialogue, obtain the first attack prompt related to the attack task, and send the first attack prompt to the target model in the first dialogue round.

[0084] A response receiving unit 12, configured to receive a first response content returned by the target model to the first attack prompt;

[0085] The semantic analysis unit 13 is used to perform semantic analysis on the first response content to determine whether the first response content conforms to the target response content corresponding to the attack task;

[0086] A dialogue direction determining unit 14, configured to determine a dialogue direction with a target model based on the first response content, the first attack prompt, and an attack correlation score between the attack tasks if the first response content does not match the target response content;

[0087] Optionally, the dialogue direction determining unit 14 may include:

[0088] A relevance score determination subunit 141 is used to determine an attack relevance score based on the relevance of the first response content with the first attack prompt and the attack task if the first response content does not match the target response content;

[0089] A relevance score comparison subunit 142, for comparing the attack relevance score with a historical relevance score corresponding to a second dialogue round, where the second dialogue round is a previous dialogue round of the first dialogue round;

[0090] A restatement subunit 143, configured to determine the direction of the conversation with the target model as restatement of the first conversation turn or restatement of the second conversation turn if the attack relevance score is less than or equal to the history relevance score;

[0091] Optionally, the restatement subunit 143 is specifically configured to compare the number of restatements of the first attack prompt in the first dialogue round with a preset restatement number threshold if the attack relevance score is less than or equal to the history relevance score;

[0092] If the number of repetitions is less than a preset repetition threshold, determining the conversation direction with the target model to be a repetition of the first conversation round;

[0093] If the number of repetitions is equal to a preset repetition number threshold, the conversation direction with the target model is determined to be repetition of the second conversation turn.

[0094] Optionally, the restatement subunit 143 is specifically configured to optimize the first attack prompt using an optimization strategy if the number of restatements is less than a preset restatement number threshold, the optimization strategy including at least one of vocabulary replacement, detail addition and modification, semantic expansion, context association, structure adjustment, and complexity adjustment;

[0095] Sends an optimized first attack hint to the target model.

[0096] Optionally, the restatement subunit 143 is specifically configured to withdraw the first dialogue turn if the number of restatements is equal to a preset restatement number threshold;

[0097] Adjusting the second attack prompt corresponding to the second dialogue round based on the prompt angle;

[0098] Sends adjusted secondary attack hint to target model.

[0099] The forward subunit 144 is configured to determine the direction of the dialogue with the target model to enter a third dialogue round if the attack relevance score is greater than the history relevance score, and the third dialogue round is the next dialogue round of the first dialogue round.

[0100] Optionally, the forward subunit 144 is further configured to, in response to the conversation direction entering the third conversation turn, perform a self-increment operation on the number of conversation turns of the first conversation turn in the conversation to obtain the number of conversation turns of the third conversation turn in the conversation;

[0101] Determine the number of dialogue rounds of the third dialogue round in the dialogue as the current number of dialogue rounds, and compare the current number of dialogue rounds with a preset number of rounds threshold;

[0102] If the current dialogue round number is less than or equal to the preset round number threshold, a third attack prompt corresponding to a third dialogue round is generated based on the first dialogue round and the second dialogue round, and the third attack prompt is sent to the target model;

[0103] If the current number of dialogue rounds is greater than the preset round number threshold, it is determined that the attack task is completed.

[0104] The attack task completion unit 15 is used to determine that the attack task is completed if the first response content matches the target response content.

[0105] In the embodiment of the present application, after the attack model receives the attack task, it obtains the non-attack topic related to the attack task, builds the basic dialogue for the attack task based on the non-attack topic, obtains the attack word related to the attack task after the target model has established the initial cognition of the attack task, introduces the attack word into the basic dialogue, obtains the first attack prompt related to the attack task, and sends the first attack prompt to the target model in the first dialogue round to receive the first response content returned by the target model for the first attack prompt, and then performs semantic analysis on the first response content, and judges whether the first response content conforms to the target response content corresponding to the attack task according to the result of the semantic analysis, and then determines whether to stop the attack. If the attack continues, the dialogue direction with the target model is determined according to the attack relevance score between the attack prompt, the response content and the attack task, so that the response content output by the target model gradually approaches the target response content corresponding to the attack task. In the process of attacking the target model, through multi-round dialogue, the response content is semantically analyzed, the dialogue direction is gradually adjusted, and the attack success rate is improved, so that researchers can identify the weak links of the target model when generating the response content, which is helpful to discover the vulnerabilities of the target model and develop more effective security defense mechanisms. This method can also help researchers discover consistency issues of the target model in multi-round dialogue interactions, as well as possible loopholes when dealing with complex contexts, thereby promoting dialogue systems to more effectively handle various challenges of multi-round dialogues during design, training, and deployment.

[0106] See also Figure 6 , Figure 6 Schematic diagram of the structure of a computer device provided in an embodiment of the present application. Figure 6 As shown, the computer device 1000 may include: at least one processor 1001, such as a CPU, at least one network interface 1004, an input / output interface 1003, a memory 1005, and at least one communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage. The memory 1005 may optionally be at least one storage device located away from the aforementioned processor 1001. As shown in FIG. Figure 6 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, an input and output interface module, and a connection establishment application.

[0107] In one embodiment, the processor 1001 may be used to call a connection establishment application stored in the memory 1005, and specifically perform the following operations:

[0108] Sending a first attack prompt related to the attack task to the target model in the first dialogue round;

[0109] receiving a first response content returned by the target model to the first attack prompt;

[0110] Perform semantic analysis on the first response content to determine whether the first response content conforms to the target response content corresponding to the attack task;

[0111] If the first response content does not match the target response content, the conversation direction with the target model is determined based on the first response content, the first attack prompt, and the attack relevance score between the attack tasks;

[0112] If the first response content matches the target response content, the attack mission is determined to be completed.

[0113] Optionally, when the processor 1001 determines the dialogue direction with the target model based on the first response content, the first attack prompt, and the attack correlation score between the attack tasks if the first response content does not match the target response content, the following operations are specifically performed:

[0114] If the first response content does not match the target response content, determining an attack relevance score based on the relevance of the first response content to the first attack prompt and the attack task;

[0115] comparing the attack relevance score with a historical relevance score corresponding to a second dialogue round, where the second dialogue round is a previous dialogue round of the first dialogue round;

[0116] If the attack relevance score is less than or equal to the history relevance score, determining the conversation direction with the target model to be a restatement of the first conversation turn or a restatement of the second conversation turn;

[0117] If the attack relevance score is greater than the history relevance score, the dialogue direction with the target model is determined to enter the third dialogue round, and the third dialogue round is the next dialogue round of the first dialogue round.

[0118] Optionally, when the processor 1001 determines that the conversation direction with the target model is to restate the first conversation turn or to restate the second conversation turn if the attack relevance score is less than or equal to the history relevance score, the processor 1001 specifically performs the following operations:

[0119] If the attack relevance score is less than or equal to the historical relevance score, comparing the number of repetitions of the first attack prompt in the first dialogue round with a preset repetition number threshold;

[0120] If the number of repetitions is less than a preset repetition threshold, determining the conversation direction with the target model to be a repetition of the first conversation round;

[0121] If the number of repetitions is equal to a preset repetition number threshold, the conversation direction with the target model is determined to be repetition of the second conversation turn.

[0122] Optionally, when the processor 1001 determines that the conversation direction with the target model is to repeat the first conversation turn if the number of repetitions is less than a preset repetition number threshold, the processor 1001 specifically performs the following operations:

[0123] If the number of repetitions is less than a preset repetition number threshold, an optimization strategy is used to optimize the first attack prompt, and the optimization strategy includes at least one of vocabulary replacement, detail addition and modification, semantic expansion, context association, structure adjustment, and complexity adjustment;

[0124] Sends an optimized first attack hint to the target model.

[0125] Optionally, when the processor 1001 determines that the conversation direction with the target model is to restate the second conversation turn if the number of restatements is equal to a preset threshold number of restatements, the processor 1001 specifically performs the following operations:

[0126] If the number of repetitions is equal to the preset repetition threshold, the first dialogue round is withdrawn;

[0127] Adjusting the second attack prompt corresponding to the second dialogue round based on the prompt angle;

[0128] Sends adjusted secondary attack hint to target model.

[0129] Optionally, after executing if the attack relevance score is greater than the history relevance score, determining that the conversation direction with the target model is to enter a third conversation round, the processor 1001 further performs the following operations:

[0130] In response to the conversation direction entering the third conversation round, the conversation round number of the first conversation round in the conversation is incremented to obtain the conversation round number of the third conversation round in the conversation;

[0131] Determine the number of dialogue rounds of the third dialogue round in the dialogue as the current number of dialogue rounds, and compare the current number of dialogue rounds with a preset number of rounds threshold;

[0132] If the current dialogue round number is less than or equal to the preset round number threshold, a third attack prompt corresponding to a third dialogue round is generated based on the first dialogue round and the second dialogue round, and the third attack prompt is sent to the target model;

[0133] If the current number of dialogue rounds is greater than the preset round number threshold, it is determined that the attack task is completed.

[0134] Optionally, before executing sending a first attack prompt related to the attack task to the target model in the first dialogue round, the processor 1001 further performs the following operations:

[0135] Receive attack tasks and obtain non-attack topics related to the attack tasks;

[0136] Construct basic dialogues for attack tasks based on non-attack topics.

[0137] Optionally, when executing sending a first attack prompt related to the attack task to the target model in the first dialogue round, the processor 1001 specifically performs the following operations:

[0138] Obtain attack words related to the attack task;

[0139] The attack words are introduced into the basic dialogue to obtain the first attack prompt related to the attack task, and the first attack prompt is sent to the target model in the first dialogue round.

[0140] In the embodiment of the present application, after the attack model receives the attack task, it obtains the non-attack topic related to the attack task, builds the basic dialogue for the attack task based on the non-attack topic, obtains the attack word related to the attack task after the target model has established the initial cognition of the attack task, introduces the attack word into the basic dialogue, obtains the first attack prompt related to the attack task, and sends the first attack prompt to the target model in the first dialogue round to receive the first response content returned by the target model for the first attack prompt, and then performs semantic analysis on the first response content, and judges whether the first response content conforms to the target response content corresponding to the attack task according to the result of the semantic analysis, and then determines whether to stop the attack. If the attack continues, the dialogue direction with the target model is determined according to the attack relevance score between the attack prompt, the response content and the attack task, so that the response content output by the target model gradually approaches the target response content corresponding to the attack task. In the process of attacking the target model, through multi-round dialogue, the response content is semantically analyzed, the dialogue direction is gradually adjusted, and the attack success rate is improved, so that researchers can identify the weak links of the target model when generating the response content, which is helpful to discover the vulnerabilities of the target model and develop more effective security defense mechanisms. This method can also help researchers discover consistency issues of the target model in multi-round dialogue interactions, as well as possible loopholes when dealing with complex contexts, thereby promoting dialogue systems to more effectively handle various challenges of multi-round dialogues during design, training, and deployment.

[0141] It should be understood that the device provided in the embodiment of the present application is used to execute the above-mentioned model attack method, and thus can achieve the same effect as the above-mentioned implementation method.

[0142] In the case of an integrated unit, the device may include a processing module and a storage module. When the device is applied to a computer device, the processing module may be used to control and manage the actions of the computer device. The storage module may be used to support the computer device in executing related program codes, etc.

[0143] The processing module may be a processor or a controller, which may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc. The storage module may be a memory.

[0144] In addition, the device provided in the embodiment of the present application can specifically be a chip, a component or a module. The chip may include a connected processor and a memory; wherein the memory is used to store instructions, and when the processor calls and executes the instructions, the chip can execute a model attack method provided in the above embodiment.

[0145] An embodiment of the present application also provides a computer-readable storage medium, in which a computer program code is stored. When the computer program code is executed on a computer, the computer executes the above-mentioned related method steps to implement a model attack method provided by the above-mentioned embodiment.

[0146] This embodiment also provides a computer program product. When the computer program product runs on a computer, it enables the computer to execute the above-mentioned related steps to implement a model attack method provided by the above embodiment.

[0147] Among them, the device, computer-readable storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0148] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0149] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0150] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A model attack method, characterized in that: The method comprises: Sending a first attack prompt related to the attack task to the target model in the first dialogue round; receiving a first response content returned by the target model to the first attack prompt; Performing semantic analysis on the first response content to determine whether the first response content conforms to the target response content corresponding to the attack task; If the first response content does not conform to the target response content, determining a dialogue direction with the target model based on the first response content, the first attack prompt, and the attack relevance score between the attack tasks; If the first response content matches the target response content, it is determined that the attack task is completed.

2. The method according to claim 1, characterized in that If the first response content does not conform to the target response content, determining a dialogue direction with the target model based on the first response content, the first attack prompt, and the attack relevance score between the attack tasks, includes: If the first response content does not conform to the target response content, determining an attack relevance score based on the relevance of the first response content to the first attack prompt and the attack task; comparing the attack relevance score with a historical relevance score corresponding to a second dialogue round, where the second dialogue round is a previous dialogue round of the first dialogue round; If the attack relevance score is less than or equal to the history relevance score, determining the conversation direction with the target model to be restatement of the first conversation turn or restatement of the second conversation turn; If the attack relevance score is greater than the history relevance score, the conversation direction with the target model is determined to enter a third conversation round, and the third conversation round is the next conversation round of the first conversation round.

3. The method according to claim 2, characterized in that If the attack relevance score is less than or equal to the historical relevance score, determining the conversation direction with the target model to be restatement of the first conversation turn or restatement of the second conversation turn includes: If the attack relevance score is less than or equal to the historical relevance score, comparing the number of repetitions of the first attack prompt in the first dialogue round with a preset repetition number threshold; If the number of repetitions is less than the preset repetition number threshold, determining the conversation direction with the target model to be a repetition of the first conversation turn; If the number of repetitions is equal to the preset repetition number threshold, the conversation direction with the target model is determined to be repetition of the second conversation turn.

4. The method according to claim 3, characterized in that If the number of repetitions is less than the preset repetition number threshold, determining the conversation direction with the target model to be repetition of the first conversation turn includes: If the number of repetitions is less than the preset repetition number threshold, optimizing the first attack prompt using an optimization strategy, wherein the optimization strategy includes at least one of vocabulary replacement, detail addition and modification, semantic expansion, context association, structure adjustment, and complexity adjustment; The optimized first attack prompt is sent to the target model.

5. The method according to claim 3, characterized in that: If the number of repetitions is equal to the preset repetition number threshold, determining the conversation direction with the target model is to retell the second conversation turn, including: If the number of repetitions is equal to the preset repetition number threshold, withdrawing the first dialogue turn; adjusting the second attack prompt corresponding to the second dialogue round based on the prompt angle; The adjusted second attack prompt is sent to the target model.

6. The method according to claim 2, characterized in that If the attack relevance score is greater than the history relevance score, determining the direction of the dialogue with the target model is to enter a third dialogue round, further comprising: In response to the conversation direction entering the third conversation round, performing a self-increment operation on the conversation round number of the first conversation round in the conversation to obtain the conversation round number of the third conversation round in the conversation; Determine the number of dialogue rounds of the third dialogue round in the dialogue as the current number of dialogue rounds, and compare the current number of dialogue rounds with a preset number of rounds threshold; If the current dialogue round number is less than or equal to the preset round number threshold, generating a third attack prompt corresponding to the third dialogue round based on the first dialogue round number and the second dialogue round number, and sending the third attack prompt to the target model; If the current dialogue round number is greater than the preset round number threshold, it is determined that the attack task is completed.

7. The method according to claim 1, characterized in that Before sending the first attack prompt related to the attack task to the target model in the first dialogue round, the method further includes: Receiving an attack task, and obtaining a non-attack topic related to the attack task; Building a basic dialogue for the attack task based on the non-attack topic; The sending a first attack prompt related to the attack task to the target model in the first dialogue round includes: Acquire attack words related to the attack task; The attack word is introduced into the basic dialogue to obtain a first attack prompt related to the attack task, and the first attack prompt is sent to the target model in a first dialogue round.

8. A model attack device, characterized in that: The device comprises: A first attack unit, used to send a first attack prompt related to an attack task to a target model in a first dialogue round; a response receiving unit, configured to receive a first response content returned by the target model to the first attack prompt; A semantic analysis unit, configured to perform semantic analysis on the first response content to determine whether the first response content conforms to a target response content corresponding to the attack task; a dialogue direction determining unit, configured to determine a dialogue direction with the target model based on the first response content, the first attack prompt, and an attack correlation score between the attack tasks if the first response content does not conform to the target response content; The attack task completion unit is used to determine that the attack task is completed if the first response content matches the target response content.

9. A computer device, characterized in that: Includes processor, memory, input and output interfaces; The processor is connected to the memory and the input / output interface respectively, wherein the input / output interface is used for page interaction, the memory is used for storing program code, and the processor is used for calling the program code to execute the method according to any one of claims 1-7.

10. A computer storage medium, characterized in that: The computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the method according to any one of claims 1 to 7 is executed.

Images