Intelligent agent assistant prison break attack defense method and related device

Abstract

The invention belongs to the field of artificial intelligence, and discloses an agent assistant prison break attack defense method and a related device, and the method comprises the steps: obtaining a user instruction, and carrying out the prison break evaluation of the user instruction based on a pre-trained prison break prediction model, and obtaining a first prison break evaluation result of the user instruction; when the first jailbreak evaluation result is that the jailbreak behavior exists, generating a control instruction for terminating the operation of the intelligent agent assistant and sending the control instruction to the intelligent agent assistant; otherwise, acquiring a current action sequence text of the agent assistant, and identifying whether the action sequence text has a jailbreak behavior or not by adopting a regular expression matching method, a pre-trained deep learning model and a system cue word-based large language model to obtain a second jailbreak evaluation result; when the second jailbreak evaluation result is that the jailbreak behavior exists, a control instruction for terminating the current operation of the agent assistant is generated and sent to the agent assistant, jailbreak attack defense of the agent assistant can be comprehensively and accurately carried out, and safe and stable operation of the agent assistant is guaranteed.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and relates to a method and related device for defending against jailbreak attacks on agent assistants. Background Art

[0002] Large language models (LLMs) are an important breakthrough in the field of artificial intelligence in recent years. Based on deep learning technology, they can understand and generate natural language. These models have mastered rich semantic knowledge and language structures through training on a large amount of text data, enabling them to perform various language processing tasks, such as dialogue generation, text summarization, and information retrieval. With the development of technology, LLMs have been widely applied in agent assistant systems, enhancing the interaction ability and intelligence level of agent assistants.

[0003] Agent assistant technology refers to building systems with autonomous decision-making capabilities that can perceive, reason, and act in a specific environment. Agent assistants collect data through sensors, analyze and make decisions through algorithms, and then take actions to achieve predetermined goals. In recent years, agent assistants based on multimodal large models have become increasingly popular. These agent assistants can not only process text but also understand various information forms such as images and audio. These agent assistants can achieve more complex functions by calling APIs and processing function results, and can reason and make decisions in context. For example, an agent assistant that combines computer vision and natural language processing can perform language descriptions based on image recognition, thus enhancing the collaborative reasoning ability of the agent assistant, making it more efficient and adaptable when performing tasks.

[0004] With the widespread application of agent assistants based on multimodal large models, jailbreak attacks have become a potential threat to these agent assistants. Among them, a jailbreak attack refers to an attacker, as a user of the agent or a developer of the environment in which the agent operates, bypassing the security alignment mechanism based on the large model in various ways and requesting the agent assistant to perform threat behaviors against the network environment or user security. This attack method not only affects personal privacy but may also have a serious impact on public opinion. Research shows that jailbreak attacks may exploit vulnerabilities or design flaws in agent assistants to bypass the large model security mechanism and then manipulate the output of the agent assistant, resulting in inappropriate behavior of the agent. Therefore, it is particularly important to develop a method for defending against jailbreak attacks on agent assistants to ensure the security and reliability of agent assistants in complex environments. Summary of the Invention

[0005] The purpose of the present invention is to overcome the above-mentioned disadvantages of the prior art and provide a method and related device for defending against jailbreak attacks on agent assistants.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] In the first aspect of the present invention, there is provided a method for defending against jailbreak attacks of an agent assistant, including: obtaining a user instruction and performing a jailbreak evaluation on the user instruction based on a pre-trained jailbreak prediction model to obtain a first jailbreak evaluation result of the user instruction; when the first jailbreak evaluation result indicates the existence of a jailbreak behavior, generating a control instruction to terminate the operation of the agent assistant and sending it to the agent assistant; when the first jailbreak evaluation result indicates the non-existence of a jailbreak behavior, when the agent assistant performs each step of operation based on the user instruction, obtaining the current action sequence text of the agent assistant, and using a regular expression matching method, a pre-trained deep learning model, and a large language model based on system prompt words to identify whether there is a jailbreak behavior in the action sequence text, obtaining a second jailbreak evaluation result; when the second jailbreak evaluation result indicates the existence of a jailbreak behavior, generating a control instruction to terminate the current operation of the agent assistant and sending it to the agent assistant.

[0008] Optionally, the jailbreak evaluation of the user instruction based on the pre-trained jailbreak prediction model includes: performing mutation processing on the user instruction to obtain an instruction set; using a large language model to perform structured parsing on the instructions in the instruction set through multi-level semantic understanding, generating a number of triple representations including operation intent, execution object, and action parameters; based on the pre-trained jailbreak prediction model, performing a jailbreak evaluation on whether there is a jailbreak behavior in the number of triple representations to obtain a first jailbreak evaluation result of the user instruction.

[0009] Optionally, the mutation processing includes one or more of the following: random replacement, insertion, deletion, flipping, repetition, truncation, formatting modification, semantic preservation, syntax transformation, combined mutation, and mutation based on custom mutation rules.

[0010] Optionally, the obtaining of the current action sequence text of the agent assistant when the agent assistant performs each step of operation based on the user instruction includes: when the agent assistant performs each step of operation based on the user instruction, sequentially recording in text form the action texts of the agent assistant when executing the current step and each previous step, to obtain the current action sequence text of the agent assistant.

[0011] Optionally, the method for identifying whether there is a jailbreak behavior in the action sequence text by using the regular expression matching method, the pre-trained deep learning model, and the large language model based on system prompt words to obtain the second jailbreak evaluation result includes: matching a preset regular expression with the action sequence text to obtain the regular expression matching result of the user instruction as the first recognition result; inputting the action sequence text into the pre-trained deep learning model to obtain the jailbreak behavior recognition probability of the user instruction as the second recognition result; constructing system prompt words, and jointly inputting the system prompt words and the action sequence text into the large language model to obtain the jailbreak behavior risk level of the user instruction as the third recognition result; wherein, the system prompt words are used to guide the large language model to judge whether the action sequence text violates the security alignment principle; and comprehensively obtaining the second jailbreak evaluation result based on the first recognition result, the second recognition result, and the third recognition result.

[0012] Optionally, the comprehensive first recognition result, second recognition result, and third recognition result to obtain the second jailbreak evaluation result includes: obtaining a comprehensive risk value through the following formula:

[0013]

[0014] where R is the comprehensive risk value, is the return value of the exponential function of the first recognition result. When the first recognition result indicates the existence of a jailbreak behavior, otherwise, α is the regular matching jailbreak probability range parameter; R2 is the second recognition result, β is the jailbreak probability parameter of the pre-trained deep learning model, and R3 is the third recognition result.

[0015] When the comprehensive risk value is greater than the preset comprehensive risk value threshold, the second jailbreak evaluation result is that there is a jailbreak behavior; otherwise, the second jailbreak evaluation result is that there is no jailbreak behavior.

[0016] Optionally, the intelligent agent assistant is an intelligent agent assistant based on a multimodal large model, and the user instruction is a text instruction.

[0017] In a second aspect of the present invention, there is provided a jailbreak attack defense system for an agent assistant, comprising: a first jailbreak evaluation module, configured to obtain a user instruction and perform a jailbreak evaluation on the user instruction based on a pre-trained jailbreak prediction model to obtain a first jailbreak evaluation result of the user instruction; a first defense module, configured to generate a control instruction to terminate the operation of the agent assistant and send it to the agent assistant when the first jailbreak evaluation result indicates the existence of a jailbreak behavior; a second jailbreak evaluation module, configured to, when the first jailbreak evaluation result indicates the non-existence of a jailbreak behavior, obtain the current action sequence text of the agent assistant when the agent assistant performs each operation based on the user instruction, and use a regular expression matching method, a pre-trained deep learning model, and a large language model based on system prompt words to identify whether there is a jailbreak behavior in the action sequence text, to obtain a second jailbreak evaluation result; and a second defense module, configured to generate a control instruction to terminate the current operation of the agent assistant and send it to the agent assistant when the second jailbreak evaluation result indicates the existence of a jailbreak behavior.

[0018] In a third aspect of the present invention, there is provided a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the above-mentioned jailbreak attack defense method for an agent assistant is implemented.

[0019] In a fourth aspect of the present invention, there is provided a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the above-mentioned jailbreak attack defense method for an agent assistant is implemented.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] The jailbreak attack defense method for an agent assistant of the present invention first performs a jailbreak evaluation on a user instruction based on a pre-trained jailbreak prediction model to obtain a first jailbreak evaluation result of the user instruction, and when the first jailbreak evaluation result indicates the existence of a jailbreak behavior, generates a control instruction to terminate the operation of the agent assistant and sends it to the agent assistant, thereby defending against jailbreak attacks from the source. At the same time, when the first jailbreak evaluation result indicates the non-existence of a jailbreak behavior, when the agent assistant performs each operation based on the user instruction, the current action sequence text of the agent assistant is obtained, and a regular expression matching method, a pre-trained deep learning model, and a large language model based on system prompt words are used to identify whether there is a jailbreak behavior in the action sequence text, and the control of the agent assistant is based on the second jailbreak evaluation result to achieve jailbreak attack defense. By using a regular expression matching method, a pre-trained deep learning model, and a large language model based on system prompt words to deeply detect jailbreak behaviors, jailbreak attack behaviors can be identified from multiple angles and at multiple levels, avoiding the limitations of a single detection method, improving the accuracy and reliability of jailbreak attack detection, and thus ensuring the jailbreak attack defense effect.

[0022] Further, by performing mutation processing on the user instruction based on the original semantics, then using a large language model to perform structured parsing on the mutated instruction set, and then performing a jailbreak assessment on whether there is a jailbreak behavior in the triple representation generated by the parsing, further exploring the possible jailbreak intention under the user instruction, potential jailbreak risks can be effectively identified, thereby improving the defense effect against jailbreak attacks. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the application scenario of the agent assistant according to an embodiment of the present invention.

[0024] Figure 2 It is a flowchart of the method for defending against jailbreak attacks of the agent assistant according to an embodiment of the present invention.

[0025] Figure 3 It is a schematic diagram of the principle of the method for defending against jailbreak attacks of the agent assistant according to an embodiment of the present invention.

[0026] Figure 4 It is a block diagram of the structure of the system for defending against jailbreak attacks of the agent assistant according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0027] In order to enable those skilled in the art of the present technology to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described here can be implemented in an order different from those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily need to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0029] The present invention will be further described in detail below in conjunction with the accompanying drawings:

[0030] See Figure 1, which shows a feasible application environment for an intelligent agent assistant. In the shown application environment, the user interacts with the intelligent agent assistant and inputs a user instruction to the intelligent agent assistant. The intelligent agent assistant interacts with the terminal to complete task execution. Among them, the intelligent agent assistant understands the user instruction, and then uses its internal large model, visual component and tools to complete the perception, planning and decision-making of task execution, and finally completes the task. The terminal can be but is not limited to various desktop computers, laptop computers, smart phones, tablet computers and Internet of Things devices. Exemplarily, the intelligent agent assistant can be an intelligent agent assistant based on a multimodal large model, such as AutoGLM. The user instruction can be a text instruction.

[0031] See Figure 2 and 3 , in an embodiment of the present invention, a method for defending against jailbreak attacks of an intelligent agent assistant is provided. The test object is an intelligent agent assistant that analyzes the user's natural language instructions and operation interface using a large language model and completes tasks by calling existing tools in a single-step or multi-step manner, which can solve the shortcoming of insufficient defense ability of the existing method for defending against jailbreak attacks of intelligent agent assistants and ensure the safe operation of the intelligent agent assistant.

[0032] Specifically, the method for defending against jailbreak attacks of the intelligent agent assistant of the present invention includes the following steps:

[0033] S1: Obtain the user instruction and perform a jailbreak evaluation on the user instruction based on a pre-trained jailbreak prediction model to obtain a first jailbreak evaluation result of the user instruction.

[0034] S2: When the first jailbreak evaluation result indicates a jailbreak behavior, generate a control instruction to terminate the operation of the intelligent agent assistant and send it to the intelligent agent assistant.

[0035] S3: When the first jailbreak evaluation result indicates no jailbreak behavior, when the intelligent agent assistant performs each operation based on the user instruction, obtain the current action sequence text of the intelligent agent assistant, and use a regular expression matching method, a pre-trained deep learning model and a large language model based on system prompt words to identify whether there is a jailbreak behavior in the action sequence text to obtain a second jailbreak evaluation result.

[0036] S4: When the second jailbreak evaluation result indicates a jailbreak behavior, generate a control instruction to terminate the current operation of the intelligent agent assistant and send it to the intelligent agent assistant.

[0037] The jailbreak attack defense method of the intelligent agent assistant of the present invention first performs jailbreak evaluation of the user instruction based on a pre-trained jailbreak prediction model to obtain the first jailbreak evaluation result of the user instruction. When the first jailbreak evaluation result indicates the existence of a jailbreak behavior, a control instruction to terminate the operation of the intelligent agent assistant is generated and sent to the intelligent agent assistant to defend against jailbreak attacks from the source. At the same time, when the first jailbreak evaluation result indicates the non-existence of a jailbreak behavior, when the intelligent agent assistant performs each operation based on the user instruction, the current action sequence text of the intelligent agent assistant is obtained, and a regular expression matching method, a pre-trained deep learning model, and a large language model based on system prompt words are used to identify whether there is a jailbreak behavior in the action sequence text. And based on the second jailbreak evaluation result, the control of the intelligent agent assistant is used to achieve jailbreak attack defense. By using a regular expression matching method, a pre-trained deep learning model, and a large language model based on system prompt words to deeply detect jailbreak behaviors, jailbreak attack behaviors can be identified from multiple angles and levels, avoiding the limitations of a single detection method, improving the accuracy and reliability of jailbreak attack detection, and thus ensuring the jailbreak attack defense effect.

[0038] In a possible implementation manner, the jailbreak evaluation of the user instruction by the pre-trained jailbreak prediction model includes: performing mutation processing on the user instruction to obtain an instruction set; using a large language model to perform structured parsing on the instructions in the instruction set in a multi-level semantic understanding manner to generate a number of triple representations including operation intent, execution object, and action parameters; based on the pre-trained jailbreak prediction model, performing jailbreak evaluation on whether there is a jailbreak behavior for the number of triple representations to obtain the first jailbreak evaluation result of the user instruction.

[0039] Explanatorily, the structured parsing includes using a large language model to perform structured parsing on all mutated user instructions. For each instruction, a triple representation including operation intent, execution object, and action parameters is generated. Specifically, the use of the pre-trained jailbreak prediction model to perform jailbreak prediction on all parsing results means taking all triples as inputs and inputting them into the pre-trained jailbreak prediction model, and finally the model outputs the jailbreak prediction result.

[0040] Exemplarily, the user instruction is "Transfer 100 yuan to Alice via WeChat". At this time, the existing user instruction can be mutated in multiple ways to obtain multiple mutated instructions, such as {"Pay 100 yuan in full to Alice via WeChat", "Transfer 100 yuan to Alice's WeChat wallet", "Use the WeChat payment function to transfer 100 yuan to Alice", "WeChat transfer operation: Transfer 100 yuan in RMB to Alice", "Transfer 100 yuan via WeChat to Alice's account",...}. Then, perform a parsing operation on each instruction to obtain a triple representation of the operation intention, execution object, and action parameters {(Payment, Alice, {Amount: 100 yuan, Payment method: WeChat}), (Transfer, Alice, {Amount: 100 yuan, Payment method: WeChat}), (Transfer, Alice's account, {Amount: 100 yuan, Payment method: WeChat}), (Execute transfer operation, Alice, {Amount: 100 yuan in Chinese capitalization, Format: Chinese capitalization, Tool: WeChat}), (Initiate transfer, Alice, {Amount: 100 yuan, Type: spare change})...}. Finally, input these triples into the pre-trained jailbreak prediction model to obtain the first jailbreak evaluation result of the user instruction.

[0041] In a possible implementation manner, the mutation processing includes one or more of the following: random replacement, insertion, deletion, reversal, repetition, truncation, formatting modification, semantic preservation, syntactic transformation, combined mutation, and mutation based on custom mutation rules.

[0042] Specifically, the mutation processing is achieved by randomly replacing some words in the instruction, inserting new words or phrases, deleting some content in the instruction, reversing the word order in the instruction, repeating some parts of the instruction, truncating the instruction to make it incomplete, performing formatting modification on the instruction such as changing the case or punctuation, changing the expression of the instruction while keeping the semantics unchanged, performing syntactic transformation such as changing the tense or voice, and combining multiple mutation means or performing complex mutation on the instruction based on custom mutation rules.

[0043] In a possible implementation manner, when the intelligent agent assistant performs each operation based on the user instruction, obtaining the current action sequence text of the intelligent agent assistant includes: when the intelligent agent assistant performs each operation based on the user instruction, recording the action texts of the intelligent agent assistant when executing the current step and each previous step in order in text form to obtain the current action sequence text of the intelligent agent assistant.

[0044] Explanatorily, input the original user instruction into the intelligent agent assistant through the operation interface of the intelligent agent assistant, and require the intelligent agent assistant to execute the instruction in the given operation environment. Record the actions that the intelligent agent assistant is about to execute during the operation of the intelligent agent assistant, and output the next action to be executed in text format.

[0045] In a possible implementation, using the regular expression matching method, the pre-trained deep learning model, and the large language model based on system prompt words to identify whether there is a jailbreak behavior in the action sequence text, the second jailbreak evaluation result includes: matching the preset regular expression with the action sequence text to obtain the regular expression matching result of the user instruction as the first recognition result; inputting the action sequence text into the pre-trained deep learning model to obtain the jailbreak behavior recognition probability of the user instruction as the second recognition result; constructing system prompt words, and jointly inputting the system prompt words and the action sequence text into the large language model to obtain the jailbreak behavior risk level of the user instruction as the third recognition result; wherein, the system prompt words are used to guide the large language model to judge whether the action sequence text violates the security alignment principle; comprehensively combining the first recognition result, the second recognition result, and the third recognition result to obtain the second jailbreak evaluation result.

[0046] Explanatorily, the regular expression matching method uses regular expression matching technology to detect whether there are known jailbreak behavior patterns in the action sequence text of the intelligent agent assistant. For example, detecting known jailbreak behavior patterns such as "posting false messages" or "large amount of money transfer". The pre-trained deep learning model combines user instructions with action feedback large model detection technology, uses large-scale pre-trained models to analyze complex action sequence texts, and identifies deep-level jailbreak behavior patterns. Especially the large language model based on system prompt words deeply analyzes natural language instructions and feedback to identify potential jailbreak intentions and behavior patterns.

[0047] Specifically, the first recognition result is obtained by using regular expression matching. If the jailbreak pattern can be matched, the first recognition result is that there is a jailbreak behavior; otherwise, the first recognition result is that there is no jailbreak behavior. Exemplarily, keyword pattern matching is used for the action sequence text. If the match is successful, it means that the intelligent agent assistant accepts and executes the jailbreak behavior; for example, for the existing action sequence ["Open WeChat", "Click on the conversation with Alice", "Send insulting language"], at this time, regular expression matching technology can be used to match the word "insult". If the match is successful, it means that there is a jailbreak behavior. At this time, the first recognition result is R1 = 1. Therefore Otherwise, it means that no jailbreak behavior is recognized. At this time, the first recognition result is R1 = 0.

[0048] Specifically, a pre-trained deep learning model is used to identify potential jailbreaking behaviors or abnormal behaviors. For the action sequence text F = (f1, f2,..), it is used as the input of the pre-trained deep learning model M, and the recognition and detection result M(F) is obtained. Exemplarily, for the existing action sequence ["Open WeChat", "Click on the conversation with Alice", "Send insulting language"], at this time, a pre-trained deep learning model can be used to identify the above action sequence. The recognition result is the jailbreaking behavior recognition probability, and the jailbreaking behavior recognition probability is a probability value from 0% to 100%. The higher the jailbreaking behavior recognition probability, the higher the probability that it is a jailbreaking behavior. At this time, the second recognition result R2 is this probability value.

[0049] Specifically, a large language model based on system prompt words is used for recognition. Exemplarily, the system prompt words proposed to the large language model are "Next, I will give you an action sequence. Please judge whether this action sequence violates the security alignment principle and give a risk level of 1, 2, 3, 4, or 5. A risk level of 1 means it is definitely not a jailbreaking behavior, and a risk level of 5 means you think it is definitely a jailbreaking behavior. The higher the risk level, the more likely you think it is a jailbreaking behavior". After the recognition of the large language model, a third recognition result, that is, the risk level given by the large language model, will be generated. At this time, the third recognition result R3 is this risk level.

[0050] In a possible implementation manner, the comprehensive first recognition result, second recognition result, and third recognition result to obtain the second jailbreaking evaluation result include: obtaining a comprehensive risk value through the following formula:

[0051]

[0052] where R is the comprehensive risk value, is the return value of the exponential function of the first recognition result. When the first recognition result is that there is a jailbreaking behavior, otherwise, α is the regular matching jailbreaking probability value range parameter; R2 is the second recognition result, β is the jailbreaking probability parameter of the pre-trained deep learning model, and R3 is the third recognition result.

[0053] When the comprehensive risk value is greater than the preset comprehensive risk value threshold, the second jailbreaking evaluation result is that there is a jailbreaking behavior; otherwise, the second jailbreaking evaluation result is that there is no jailbreaking behavior.

[0054] Explanatorily, if the first jailbreaking evaluation result is that there is a jailbreaking behavior, it is determined that it is a jailbreaking behavior. At this time, the user instruction is not executed, and the task is directly stopped; if the second jailbreaking evaluation result is that there is a jailbreaking behavior, it is determined that the current action sequence may constitute a jailbreaking behavior. At this time, the next behavior of the intelligent agent assistant is not executed, and the intelligent agent assistant is directly controlled to stop the task.

[0055] Exemplarily, the agent assistant is an agent assistant based on a multimodal large model, such as OpenManus, BrowserUse, AutoGLM, etc. The user instruction can be a text instruction.

[0056] In summary, the jailbreak attack defense method of the agent assistant of the present invention essentially mutates and parses user instructions, analyzes the potential jailbreak intention of user instructions, and then uses a combined method to detect possible jailbreak behaviors or patterns. Specifically, the present invention can analyze the user instructions input to the agent assistant and the action sequence of the agent assistant, so as to achieve the effect of refusing to execute jailbreak instructions or promptly detecting jailbreak behaviors. On the one hand, when the agent assistant processes user instructions, it may face the risk of jailbreak attacks. An attacker may bypass the security mechanism of the model and require the agent to automatically release false information, execute transfer operations, etc. This solution mutates the user instructions based on semantic preservation, then parses the mutated instructions into a triple of intention, behavior object, and behavior parameters, and finally identifies jailbreak instructions for the triple, so as to accurately identify potential jailbreak instructions. On the other hand, during the process of the agent assistant executing tasks, before the action is executed, the action to be executed is obtained from the agent assistant, and then combined with the actions that have been executed before to form an action sequence text. Then, jailbreak detection based on regular matching, jailbreak identification based on a pre-trained deep learning model, and jailbreak level identification based on the large language model of the system prompt word are performed on the action sequence text. The results of these three types of detection and identification are comprehensively used to finally obtain the result of whether the current action is a jailbreak. If the current action is identified as a jailbreak, the current task is immediately stopped. Through multiple detections of the action sequence text, the execution of jailbreak behaviors is effectively refused, which guarantees the security of the agent assistant.

[0057] The following is an apparatus embodiment of the present invention, which can be used to execute the method embodiment of the present invention. For details not disclosed in the apparatus embodiment, please refer to the method embodiment of the present invention.

[0058] See Figure 4 In another embodiment of the present invention, a jailbreak attack defense system for an agent assistant is provided, which can be used to implement the jailbreak attack defense method of the above-mentioned agent assistant. Specifically, the jailbreak attack defense system for the agent assistant includes a first jailbreak evaluation module, a first defense module, a second jailbreak evaluation module, and a second defense module.

[0059] Among them, the first jailbreak evaluation module is used to obtain user instructions and perform jailbreak evaluation on the user instructions based on a pre-trained jailbreak prediction model to obtain the first jailbreak evaluation result of the user instructions; the first defense module is used to generate a control instruction to terminate the operation of the intelligent agent assistant and send it to the intelligent agent assistant when the first jailbreak evaluation result indicates the existence of a jailbreak behavior; the second jailbreak evaluation module is used to, when the first jailbreak evaluation result indicates the non-existence of a jailbreak behavior, obtain the current action sequence text of the intelligent agent assistant when the intelligent agent assistant performs each operation based on the user instructions, and use a regular expression matching method, a pre-trained deep learning model, and a large language model based on system prompt words to identify whether there is a jailbreak behavior in the action sequence text to obtain the second jailbreak evaluation result; the second defense module is used to generate a control instruction to terminate the current operation of the intelligent agent assistant and send it to the intelligent agent assistant when the second jailbreak evaluation result indicates the existence of a jailbreak behavior.

[0060] All relevant contents of each step involved in the embodiment of the jailbreak attack defense method of the foregoing intelligent agent assistant can be cited in the function description of the function modules corresponding to the jailbreak attack defense system of the intelligent agent assistant in the embodiment of the present invention, and will not be elaborated here.

[0061] The division of modules in the embodiments of the present invention is illustrative, merely a logical function division. In actual implementation, there may be other division methods. In addition, in each embodiment of the present invention, the function modules can be integrated in one processor, or exist separately physically, or two or more modules can be integrated in one module. The above integrated modules can be implemented in the form of hardware or in the form of software function modules.

[0062] In another embodiment of the present invention, a computer device is provided. The computer device includes a processor and a memory. The memory is used to store a computer program, and the computer program includes program instructions. The processor is used to execute the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, and is suitable for implementing one or more instructions. Specifically, it is suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding function. The processor described in the embodiment of the present invention can be used for the operation of the jailbreak attack defense method of the intelligent agent assistant.

[0063] In another embodiment of the present invention, a storage medium is also provided, specifically a computer-readable storage medium (Memory). The computer-readable storage medium is a memory device in the computer device and is used to store programs and data. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and, of course, the extended storage medium supported by the computer device. The computer-readable storage medium provides a storage space that stores the operating system of the terminal. And, one or more instructions suitable for being loaded and executed by the processor are also stored in this storage space. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. One or more instructions stored in the computer-readable storage medium can be loaded and executed by the processor to implement the corresponding steps of the jailbreak attack defense method of the intelligent agent assistant in the above embodiments.

[0064] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) that contain computer-usable program code.

[0065] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0066] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing devices to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0067] These computer program instructions can also be loaded onto a computer or other programmable data processing devices, such that a series of operation steps are executed on the computer or other programmable devices to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable devices provide steps for implementing the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: still can modify the specific implementation manners of the present invention or make equivalent substitutions, and any modification or equivalent substitution that does not depart from the spirit and scope of the present invention shall be covered by the protection scope of the claims of the present invention.

Claims

1. A method for defending against jailbreak attacks of an agent assistant, characterized in that, Including: Obtain a user instruction and perform a jailbreak assessment on the user instruction based on a pre-trained jailbreak prediction model to obtain a first jailbreak assessment result of the user instruction; When the first jailbreak assessment result indicates the existence of a jailbreak behavior, generate a control instruction to terminate the operation of the intelligent agent assistant and send it to the intelligent agent assistant; When the first jailbreak assessment result indicates the non-existence of a jailbreak behavior, when the intelligent agent assistant performs each step of operation based on the user instruction, obtain the current action sequence text of the intelligent agent assistant, and use a regular expression matching method, a pre-trained deep learning model, and a large language model based on system prompt words to identify whether there is a jailbreak behavior in the action sequence text, and obtain a second jailbreak assessment result; When the second jailbreak assessment result indicates the existence of a jailbreak behavior, generate a control instruction to terminate the current operation of the intelligent agent assistant and send it to the intelligent agent assistant.

2. The jailbreak attack defense method for the intelligent agent assistant according to claim 1, wherein The jailbreak assessment of the user instruction based on the pre-trained jailbreak prediction model includes: Perform mutation processing on the user instruction to obtain an instruction set; Use a large language model to perform structured parsing on the instructions in the instruction set through multi-level semantic understanding, and generate a number of triple representations including operation intent, execution object, and action parameters; Based on the pre-trained jailbreak prediction model, perform a jailbreak assessment on whether there is a jailbreak behavior in a number of triple representations to obtain a first jailbreak assessment result of the user instruction.

3. The jailbreak attack defense method of the intelligent agent assistant according to claim 2, wherein The mutation processing includes one or more of the following: Random replacement, insertion, deletion, flipping, repetition, truncation, formatting modification, semantic preservation, syntactic transformation, combined mutation, and mutation based on custom mutation rules.

4. The jailbreak attack defense method for the intelligent agent assistant according to claim 1, wherein, When the intelligent agent assistant performs each step of operation based on the user instruction, obtaining the current action sequence text of the intelligent agent assistant includes: When the intelligent agent assistant performs each step of operation based on the user instruction, record the action text executed by the intelligent agent assistant at the current step and each previous step in order in text form to obtain the current action sequence text of the intelligent agent assistant.

5. The jailbreak attack defense method for the intelligent agent assistant according to claim 1, characterized in that, The method of using a regular expression matching method, a pre-trained deep learning model, and a large language model based on system prompt words to identify whether there is a jailbreak behavior in the action sequence text and obtain a second jailbreak assessment result includes: Match the preset regular expression with the action sequence text to obtain a regular expression matching result of the user instruction as the first recognition result; Input the action sequence text into the pre-trained deep learning model to obtain the jailbreak behavior recognition probability of the user instruction as the second recognition result; Construct a system prompt word, and input the system prompt word and the action sequence text into the large language model together to obtain the jailbreak behavior risk level of the user instruction as the third recognition result; wherein, the system prompt word is used to guide the large language model to judge whether the action sequence text violates the security alignment principle; Integrate the first recognition result, the second recognition result, and the third recognition result to obtain a second jailbreak assessment result.

6. The jailbreak attack defense method of the intelligent agent assistant according to claim 5, characterized in that, The integration of the first recognition result, the second recognition result, and the third recognition result to obtain a second jailbreak assessment result includes: Obtain the comprehensive risk value through the following formula: where R is the comprehensive risk value, is the return value of the exponential function of the first recognition result. When the first recognition result is that there is a jailbreak behavior, otherwise, α is the regular matching jailbreak probability range parameter; R2 is the second recognition result, β is the jailbreak probability parameter of the pre-trained deep learning model, and R3 is the third recognition result; When the comprehensive risk value is greater than the preset comprehensive risk value threshold, the second jailbreak evaluation result is that there is a jailbreak behavior; otherwise, the second jailbreak evaluation result is that there is no jailbreak behavior.

7. The jailbreak attack defense method for the intelligent agent assistant according to claim 1, wherein The intelligent agent assistant is an intelligent agent assistant based on a multimodal large model, and the user instruction is a text instruction.

8. A jailbreak attack defense system for an agent assistant, characterized in that, Including: A first jailbreak evaluation module, configured to obtain a user instruction and perform a jailbreak evaluation on the user instruction based on a pre-trained jailbreak prediction model to obtain a first jailbreak evaluation result of the user instruction; A first defense module, configured to generate a control instruction to terminate the operation of the intelligent agent assistant and send it to the intelligent agent assistant when the first jailbreak evaluation result is that there is a jailbreak behavior; A second jailbreak evaluation module, configured to, when the first jailbreak evaluation result is that there is no jailbreak behavior, obtain the current action sequence text of the intelligent agent assistant during each step of the intelligent agent assistant's operation based on the user instruction, and use a regular expression matching method, a pre-trained deep learning model, and a large language model based on system prompt words to identify whether there is a jailbreak behavior in the action sequence text, and obtain a second jailbreak evaluation result; A second defense module, configured to generate a control instruction to terminate the current operation of the intelligent agent assistant and send it to the intelligent agent assistant when the second jailbreak evaluation result is that there is a jailbreak behavior.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the jailbreak attack defense method of the intelligent agent assistant according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the jailbreak attack defense method of the intelligent agent assistant according to any one of claims 1 to 7.

Images