Large language model instruction risk detection method and device, equipment and storage medium

Abstract

The invention discloses a large language model instruction risk detection method and device, equipment and a storage medium, and aims to enhance the detection and identification capability of risk instructions by performing legality verification, domain constraint detection, baseline behavior detection, instruction role detection and other systematic verification on the current instruction of a large language model. Meanwhile, the comprehensiveness and the accuracy of detection are also improved; besides, corresponding detection results can be obtained in each detection process, so that the detection process is more transparent and convenient to analyze and understand; moreover, each detection process is hierarchical, and the current instruction which does not pass the previous detection is directly rejected, so that the detection efficiency is higher.

Description

Technical Field

[0001] The present application relates to the field of network security technology, and in particular to a method, device, equipment and storage medium for detecting risks of large language model instructions. Background Art

[0002] With the popularity of Large Language Model (LLM), both general large language models and domain large language models are widely used, but they also cause many security issues. Since the application scenarios and application fields of large language models vary greatly, it is very important to identify risks and protect security before, during and after the interaction of large language models.

[0003] Traditionally, when detecting risks of large language model instructions, the technologies that are currently used and researched are mostly concentrated on the protection of the large language model itself, as well as the legality and compliance verification of the large language model instructions. Most of them are scattered and independent detection and protection methods, lacking systematic detection and protection solutions and mechanisms.

[0004] Therefore, the above-mentioned large language model instruction risk detection method has problems such as being incomplete and having low accuracy. Summary of the invention

[0005] The present application aims to at least solve the technical problems existing in the prior art. To this end, the present application proposes a method for detecting the risk of a large language model instruction in a first aspect. The method comprises:

[0006] Performing a legality check on the current instruction of the large language model to generate a first detection result;

[0007] When the current instruction is determined to be legal based on the first detection result, a domain constraint detection is performed on the current instruction to generate a second detection result; wherein the second detection result is used to indicate whether the current instruction exceeds a corresponding preset constraint condition, and the preset constraint condition is determined based on the attribute information, application domain information, and application scenario information of the large language model;

[0008] When the second detection result is that the detection is passed, a baseline behavior detection is performed on the current instruction to generate a third detection result; wherein the third detection result is used to indicate whether the current instruction is a normal behavior;

[0009] When the third detection result is that the detection is passed, the current instruction is subjected to instruction role detection to generate a fourth detection result, and when the fourth detection result is that the detection is passed, the large language model is interacted based on the current instruction; wherein the fourth detection result is used to indicate whether the current instruction is consistent with the current user role.

[0010] In a possible implementation, the legality check includes a legality compliance check and an instruction injection check, and the legality check is performed on the current instruction to generate a first detection result, including:

[0011] A preset pattern matching algorithm is used to perform legality and compliance verification and instruction injection verification on the current instruction to generate a first detection result; wherein, the legality and compliance verification is used to determine whether the current instruction includes illegal content, and the instruction injection verification is used to determine whether the current instruction is other instructions that cover or bypass expected instructions or preset constraints.

[0012] In a possible implementation, performing domain constraint detection on the current instruction to generate a second detection result includes:

[0013] Perform type judgment on the current instruction and generate a type judgment result;

[0014] Based on the type judgment result and the preset constraint conditions corresponding to the preset classification category, a second detection result is generated.

[0015] In a possible implementation, performing a baseline behavior detection on the current instruction to generate a third detection result includes:

[0016] Obtaining a preset baseline detection sub-model; wherein the preset baseline detection sub-model is trained based on a sample data set, and the sample data set includes a plurality of historical instruction information, system role setting information, and system interaction information;

[0017] A baseline behavior pattern detection is performed on the current instruction through a preset baseline detection sub-model to generate a third detection result.

[0018] In a possible implementation, the process of constructing the preset baseline detection sub-model includes:

[0019] Obtain a sample data set and perform data processing on the sample data set to generate a target data set; wherein the data processing includes data preprocessing, standardization processing, word segmentation processing and encoding processing;

[0020] Perform clustering on the target data set, and perform behavioral pattern recognition on the clustering results to obtain recognition results; the recognition results include normal behavior patterns or abnormal behavior patterns, and the normal behavior patterns are used to construct baseline behavior patterns;

[0021] The initial baseline detection sub-model is trained using data corresponding to normal behavior patterns in the target data set to generate a preset baseline detection sub-model.

[0022] In a possible implementation, performing instruction role detection on the current instruction to generate a fourth detection result includes:

[0023] Perform intention recognition and standardization processing on the current instruction to obtain the processing result;

[0024] Using a preset similarity algorithm to calculate the similarity between the processing result and each instruction in the preset instruction data set;

[0025] Based on the maximum value of the similarity, determine the target instruction level corresponding to the current instruction;

[0026] The target user role corresponding to the target instruction level is compared with the current user role corresponding to the current instruction, and a fourth detection result is generated according to the comparison result.

[0027] In a possible implementation, the method further includes:

[0028] When any one of the first detection result, the second detection result, the third detection result and the fourth detection result fails the detection, an error message is generated and the interaction process corresponding to the current instruction is terminated.

[0029] In a second aspect, the present application provides a device for detecting large language model instruction risks, the device comprising:

[0030] A first generating module, used to perform a legality check on the current instruction of the large language model and generate a first detection result;

[0031] A second generation module is used to perform a domain constraint detection on the current instruction and generate a second detection result when the current instruction is determined to be legal based on the first detection result; wherein the second detection result is used to indicate whether the current instruction exceeds a corresponding preset constraint condition, and the preset constraint condition is determined based on the attribute information, application domain information and application scenario information of the large language model;

[0032] A third generating module is used to perform a baseline behavior detection on the current instruction and generate a third detection result when the second detection result is a passed detection; wherein the third detection result is used to indicate whether the current instruction is a normal behavior;

[0033] The fourth generation module is used to perform instruction role detection on the current instruction when the third detection result is a passed detection, generate a fourth detection result, and interact with the large language model based on the current instruction when the fourth detection result is a passed detection; wherein the fourth detection result is used to indicate whether the current instruction is consistent with the current user role.

[0034] In a possible implementation manner, the first generating module is specifically used to:

[0035] A preset pattern matching algorithm is used to perform legality and compliance verification and instruction injection verification on the current instruction to generate a first detection result; wherein, the legality and compliance verification is used to determine whether the current instruction includes illegal content, and the instruction injection verification is used to determine whether the current instruction is other instructions that cover or bypass expected instructions or preset constraints.

[0036] In a possible implementation manner, the second generating module is specifically used for:

[0037] Perform type judgment on the current instruction and generate a type judgment result;

[0038] Based on the type judgment result and the preset constraint conditions corresponding to the preset classification category, a second detection result is generated.

[0039] In a possible implementation manner, the third generating module is specifically used to:

[0040] Obtaining a preset baseline detection sub-model; wherein the preset baseline detection sub-model is trained based on a sample data set, and the sample data set includes a plurality of historical instruction information, system role setting information, and system interaction information;

[0041] A baseline behavior pattern detection is performed on the current instruction through a preset baseline detection sub-model to generate a third detection result.

[0042] In a possible implementation manner, the third generating module is further used to:

[0043] Obtain a sample data set and perform data processing on the sample data set to generate a target data set; wherein the data processing includes data preprocessing, standardization processing, word segmentation processing and encoding processing;

[0044] Perform clustering on the target data set, and perform behavioral pattern recognition on the clustering results to obtain recognition results; the recognition results include normal behavior patterns or abnormal behavior patterns, and the normal behavior patterns are used to construct baseline behavior patterns;

[0045] The initial baseline detection sub-model is trained using data corresponding to normal behavior patterns in the target data set to generate a preset baseline detection sub-model.

[0046] In a possible implementation manner, the fourth generating module is specifically used to:

[0047] Perform intention recognition and standardization processing on the current instruction to obtain the processing result;

[0048] Using a preset similarity algorithm to calculate the similarity between the processing result and each instruction in the preset instruction data set;

[0049] Based on the maximum value of the similarity, determine the target instruction level corresponding to the current instruction;

[0050] The target user role corresponding to the target instruction level is compared with the current user role corresponding to the current instruction, and a fourth detection result is generated according to the comparison result.

[0051] In a possible implementation manner, the above-mentioned large language model instruction risk detection device is further used for:

[0052] When any one of the first detection result, the second detection result, the third detection result and the fourth detection result fails the detection, an error message is generated and the interaction process corresponding to the current instruction is terminated.

[0053] The third aspect of the present application proposes an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the large language model instruction risk detection method as described in the first aspect.

[0054] In a fourth aspect, the present application proposes a computer-readable storage medium, in which at least one instruction, at least one program, a code set or an instruction set is stored. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the large language model instruction risk detection method as described in the first aspect.

[0055] The embodiments of the present application have the following beneficial effects:

[0056] The method for detecting the risk of large language model instructions provided in the embodiment of the present application includes: performing a legality check on the current instruction of the large language model to generate a first detection result, and when the current instruction is determined to be legal based on the first detection result, performing a domain constraint detection on the current instruction to generate a second detection result, and when the second detection result is a detection pass, performing a baseline behavior detection on the current instruction to generate a third detection result, and when the third detection result is a detection pass, performing an instruction role detection on the current instruction to generate a fourth detection result, and when the fourth detection result is a detection pass, performing an interaction of the large language model based on the current instruction. This scheme enhances the detection and identification capabilities of risky instructions by performing systematic verifications such as legality verification, domain constraint detection, baseline behavior detection, and instruction role detection on the current instruction of the large language model, and also improves the comprehensiveness and accuracy of the detection; in addition, each detection process can obtain corresponding detection results, making the detection process more transparent and easy to analyze and understand; and each detection process is hierarchical, and the current instruction that fails the previous detection is directly rejected, making the detection more efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A block diagram of a computer device provided in an embodiment of the present application;

[0058] Figure 2 A flowchart of a method for detecting risk of large language model instructions provided in an embodiment of the present application;

[0059] Figure 3 A flowchart of the steps for generating a second detection result provided in an embodiment of the present application;

[0060] Figure 4 A flowchart of the steps for generating a third detection result provided in an embodiment of the present application;

[0061] Figure 5 A flowchart of the steps of constructing a preset baseline detection sub-model provided in an embodiment of the present application;

[0062] Figure 6 A flowchart of the steps for generating a fourth detection result provided in an embodiment of the present application;

[0063] Figure 7 A structural block diagram of a large language model instruction risk detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0065] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, "multiple" means two or more. In addition, the use of "based on" or "according to" means openness and inclusiveness, because the process, steps, calculations or other actions "based on" or "according to" one or more of the conditions or values ​​may be based on additional conditions or values ​​beyond the described values ​​in practice.

[0066] The large language model instruction risk detection method provided in the present application can be applied to a computer device (electronic device), which can be a server or a terminal, wherein the server can be a single server or a server cluster composed of multiple servers. The embodiments of the present application do not specifically limit this. The terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices.

[0067] Take the computer device as a server as an example. Figure 1 A block diagram of a server is shown, such as Figure 1 As shown, the server may include a processor and a memory connected via a system bus. The processor of the server is used to provide computing and control capabilities. The memory of the server includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. When the computer program is executed by the processor, a method for detecting the risk of a large language model instruction is implemented.

[0068] Those skilled in the art will understand that Figure 1 The structure shown in the figure is only a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the server to which the solution of the present application is applied. Optionally, the server may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0069] It should be noted that the execution subject of the embodiments of the present application may be a computer device or a detection device for large language model instruction risks. The following method embodiments will be described using a computer device as the execution subject.

[0070] Figure 2 A flowchart of a method for detecting large language model instruction risks provided by an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0071] Step 202: Perform a legality check on the current instruction of the large language model to generate a first detection result.

[0072] Among them, Large Language Model (LLM) refers to a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text. Large language model can handle a variety of natural language tasks, such as text classification, question answering, dialogue, etc., and is an important path to artificial intelligence.

[0073] In order to address the problems of incomplete, inaccurate, unsystematic and insufficiently transparent security checks on the current instructions of the large language model before model interaction, after obtaining the current instructions of the large language model, the legality of the current instructions can be checked to generate a first test result.

[0074] In some optional embodiments, the legality check may include a legality and compliance check and an instruction injection check, so that the legality check of the current instruction is performed to generate a first detection result, including: using a preset pattern matching algorithm to perform a legality and compliance check and an instruction injection check on the current instruction to generate a first detection result. The legality and compliance check is used to determine whether the current instruction includes illegal content, and the instruction injection check is used to determine whether the current instruction is another instruction that covers or bypasses an expected instruction or a preset constraint condition.

[0075] Optionally, when performing legality and compliance verification, a pattern matching method may be used to perform a legality verification on the current instruction input, that is, to verify whether the current instruction input includes illegal content or hidden illegal content.

[0076] Optionally, when performing command injection verification, command injection refers to the user's attempt to manipulate the artificial intelligence system by providing input to override or bypass the expected instructions or preset constraints set by the developer. There are two common command injection strategies, one is to use delimiters and clear instructions in system messages, and the other is to add additional prompts to ask the user whether to attempt command injection. For these two strategies, pattern matching can be used to match keywords, such as "ignore", "please confirm" and other keywords.

[0077] If at least one of the above two matching results is a successful match, it can be considered that the current instruction is illegal, that is, the first detection result can be obtained.

[0078] Step 204: When it is determined that the current instruction is legal based on the first detection result, a domain constraint detection is performed on the current instruction to generate a second detection result.

[0079] Optionally, additional preset constraints may be added based on the attribute information, application domain information, and application scenario information of the large language model. For example, for a large language model of a domain class, relevant domain constraints may be added, and interaction is prohibited if the domain constraints are exceeded.

[0080] In some optional embodiments, Figure 3 As shown, Figure 3 A flowchart of the steps of generating a second detection result provided in an embodiment of the present application includes:

[0081] Step 302: Perform type judgment on the current instruction and generate a type judgment result.

[0082] Step 304: Generate a second detection result based on the type judgment result and the preset constraint conditions corresponding to the preset classification category.

[0083] Among them, some classification tools can be used in advance to classify the intent of the instructions of the large language model. Optionally, the classification tool can be an open source small model or an intelligent algorithm, and the intelligent algorithm can be a classification algorithm. Then, the type of instruction can be determined. For example, for a medical large language model, only instructions related to medical care and health are supported, and the rest are prohibited from interaction, that is, different types of large language models will have corresponding types of instructions.

[0084] In addition, for illegal or unethical taboos in specific industries, such as medical large language models, additional preset constraints should be added, such as refusing to judge the gender of an infant based on information about an unborn baby or providing guidance on professional methods for testing gender. In this way, large language models for different fields can be adapted by optimizing only the domain constraint detection part in the later stage, making them have strong migration and generalization capabilities.

[0085] Therefore, when the current instruction is determined to be legal based on the first detection result, when performing domain constraint detection on the current instruction, the type of the current instruction can be judged first to generate a type judgment result, and then based on the type judgment result and the preset constraint conditions corresponding to the preset classification category, a second detection result can be generated.

[0086] Among them, the preset classification categories of the instructions correspond to different types of large language models, and different types of large language models also correspond to corresponding preset constraints. Therefore, the second detection result can be generated based on the type judgment result and the preset constraints corresponding to the preset classification categories. The second detection result is used to indicate whether the current instruction exceeds the corresponding preset constraints, and the preset constraints are determined based on the attribute information, application field information and application scenario information of the large language model.

[0087] Step 206: When the second detection result is that the detection is passed, a baseline behavior detection is performed on the current instruction to generate a third detection result.

[0088] Among them, in some optional embodiments, such as Figure 4 As shown, Figure 4 A flowchart of the steps of generating a third detection result provided in an embodiment of the present application includes:

[0089] Step 402: Obtain a preset baseline detection sub-model.

[0090] Step 404: Perform baseline behavior pattern detection on the current instruction through a preset baseline detection sub-model to generate a third detection result.

[0091] The preset baseline detection sub-model is trained based on a sample data set, and the sample data set may include a plurality of historical instruction information, system role setting information, and system interaction information. Optionally, the system interaction information may include information such as interaction time and frequency.

[0092] The preset baseline detection submodel needs to be pre-built, optionally, such as Figure 5 As shown, Figure 5 A flowchart of the steps of constructing a preset baseline detection sub-model provided in an embodiment of the present application includes:

[0093] Step 502: Obtain a sample data set and perform data processing on the sample data set to generate a target data set.

[0094] Step 504: cluster the target data set, and perform behavioral pattern recognition on the clustering results to obtain recognition results.

[0095] Step 506: Train the initial baseline detection sub-model using data corresponding to the normal behavior pattern in the target data set to generate a preset baseline detection sub-model.

[0096] Among them, data processing may include data preprocessing, standardization processing, word segmentation processing and encoding processing. Specifically, data preprocessing and standardization processing in data processing can remove invalid records, records with obvious errors in interaction, records where users clearly feedback that the information is wrong, incomplete, abnormal terminals, etc.

[0097] Optionally, the standardized data can be segmented using technologies such as Jieba and HanLP to extract key information. The extracted key information can then be encoded using methods such as One-Hot encoding, bag-of-words model, TF-IDF, etc., so that the key information can be converted into numerical features, that is, the target data set is obtained.

[0098] Next, the target data set can be clustered, and the behavior pattern recognition can be performed on the clustering results to obtain the recognition results. When clustering, K-means clustering, second-order clustering, system clustering and other algorithms can be used to cluster the target data set after the above encoding processing. When selecting the clustering algorithm, the corresponding algorithm can be selected according to the current environment, resources, experience and preferences of model experts.

[0099] Optionally, experts can determine whether each type of behavior after clustering is a normal behavior pattern or an abnormal behavior pattern based on experience. Because the clustered data is of a limited number of types, it is only necessary to determine whether each type is a normal behavior pattern or an abnormal behavior pattern. Thus, the recognition result can include a normal behavior pattern or an abnormal behavior pattern, and the normal behavior pattern is used to construct a baseline behavior pattern.

[0100] Optionally, when determining the classification algorithm used in pattern recognition, a suitable machine learning algorithm can be selected based on the experience and preferences of the model expert and other factors such as resource limitations and environmental requirements, which may be a decision tree, random forest, support vector machine, neural network or other classification algorithm.

[0101] Next, the initial baseline detection sub-model can be trained using the data corresponding to the normal behavior pattern in the target data set to generate a preset baseline detection sub-model. Optionally, the performance of the model can be evaluated using methods such as cross-validation to check the accuracy, recall rate, and F1 score of the model in identifying normal and abnormal behaviors, and the model parameters can be adjusted based on the evaluation results to improve the model performance. In addition, after the model indicators are met, the model can be parameterized for subsequent use. After parameterization, the preset baseline detection sub-model can directly identify whether the current instruction belongs to the user's normal behavior, and give a probability value that it conforms to the normal behavior pattern.

[0102] Finally, the baseline behavior pattern detection can be performed on the current instruction through the preset baseline detection sub-model to detect potential abnormal user behavior, and obtain whether the current instruction is a normal behavior and the probability value of belonging to the normal behavior pattern, that is, to obtain the third detection result, which is used to indicate whether the current instruction is a normal behavior.

[0103] Step 208: When the third detection result is that the detection is passed, perform instruction role detection on the current instruction to generate a fourth detection result, and when the fourth detection result is that the detection is passed, interact with the large language model based on the current instruction.

[0104] Among them, when performing instruction role detection, it is necessary to pre-specify the instruction role detection strategy. Specifically, enough instructions can be collected first, and their intents can be analyzed and formatted and standardized to obtain a preset instruction data set, denoted as instruc_data. Then, for each instruction in the preset instruction data set, each instruction is classified according to the severity of the security risk that the instruction may cause, and the classified instruction security level matrix is ​​denoted as instruc_lv. It should be noted that this classification is generally related to the characteristics of the large language model itself, and relevant experts need to participate in the classification.

[0105] Next, enough roles can be set according to the instruction security level matrix, and each instruction security level in the instruction security level matrix can be matched to each role to form an instruction level role set, recorded as instruc_lv_roles. It should be noted that the roles here correspond to the instruction security levels, which can be a many-to-many relationship.

[0106] Then, for each role in the instruction-level role set, standards for meeting the role are established. For example, for the role of chief physician, the role requires filling out relevant information such as a doctor's license, and the role requires the ability to interact with the large language model on more sensitive topics that general user roles may not be able to handle.

[0107] Finally, the instruction security level matrix and the security level role set can be associated and initialized with roles for later use.

[0108] Therefore, when the third detection result is a pass, the current instruction can be detected according to the above instruction role detection strategy to generate a fourth detection result. That is, the instruction role strategy can be used to detect the current instruction. If it is detected that the current instruction satisfies the role, the subsequent operation is performed. Otherwise, the behavior is rejected and a prompt is given and returned.

[0109] Alternatively, if Figure 6 As shown, Figure 6 A flowchart of steps for generating a fourth detection result provided in an embodiment of the present application includes:

[0110] Step 602: perform intent recognition and standardization processing on the current instruction to obtain a processing result.

[0111] Step 604: Use a preset similarity algorithm to calculate the similarity between the processing result and each instruction in the preset instruction data set.

[0112] Step 606: Determine the target instruction level corresponding to the current instruction based on the maximum value of the similarities.

[0113] Step 608: Compare the target user role corresponding to the target instruction level with the current user role corresponding to the current instruction, and generate a fourth detection result according to the comparison result.

[0114] Among them, a command similarity threshold can be set first, recorded as simlar0, and then the current command is subjected to intent recognition and standardization processing to obtain a processing result, recorded as instruc_curr. Then, a preset similarity algorithm can be used to calculate the similarity between the processing result instruc_curr and each instruction in the preset command data set instruc_data, wherein the preset similarity algorithm can be a cosine similarity algorithm.

[0115] Based on the maximum value of the similarity, the target instruction level corresponding to the current instruction is determined, and it can be judged whether it exceeds the instruction similarity threshold simlar0. If it is greater than or equal to the instruction similarity threshold simlar0, the target instruction level corresponding to the current instruction is the instruction level corresponding to the maximum similarity, recorded as instruc_sim_level. If it is less than the instruction similarity threshold simlar0, optionally, the judgment can be recalculated.

[0116] Thus, the target user role corresponding to the target instruction level can be obtained, the target user role corresponding to the target instruction level can be compared with the current user role corresponding to the current instruction, and a fourth detection result can be generated based on the comparison result, and the fourth detection result is used to indicate whether the current instruction is consistent with the current user role. That is, if the target user role is consistent with the current user role corresponding to the current instruction, the detection is passed, and if the comparison is inconsistent, the current instruction is rejected. If the fourth detection result is that the detection is passed, the large language model can be interacted based on the current instruction.

[0117] In some optional embodiments, when any one of the first detection result, the second detection result, the third detection result and the fourth detection result fails the detection, an error message may be generated and the interaction process corresponding to the current instruction may be terminated.

[0118] The present application provides a method for detecting the risk of instructions of a large language model, the method comprising: performing a legality check on the current instruction of the large language model to generate a first detection result, performing a domain constraint check on the current instruction to generate a second detection result when the current instruction is determined to be legal based on the first detection result, performing a baseline behavior check on the current instruction to generate a third detection result when the second detection result is a pass, performing an instruction role check on the current instruction to generate a fourth detection result when the third detection result is a pass, and performing an interaction of the large language model based on the current instruction when the fourth detection result is a pass. This scheme enhances the detection and identification capabilities of risky instructions by performing systematic checks such as legality check, domain constraint check, baseline behavior check, and instruction role check on the current instruction of the large language model, while also improving the comprehensiveness and accuracy of the detection; in addition, each detection process can obtain corresponding detection results, making the detection process more transparent and easy to analyze and understand; and each detection process is hierarchical, and the current instruction that fails the previous detection is directly rejected, making the detection more efficient.

[0119] Figure 7 A structural block diagram of a large language model instruction risk detection device provided in an embodiment of the present application.

[0120] like Figure 7As shown, the large language model instruction risk detection device 700 includes:

[0121] The first generating module 702 is used to perform a legality check on the current instruction of the large language model and generate a first detection result.

[0122] The second generation module 704 is used to perform a domain constraint check on the current instruction and generate a second detection result when the current instruction is determined to be legal based on the first detection result; wherein the second detection result is used to indicate whether the current instruction exceeds the corresponding preset constraint condition, and the preset constraint condition is determined based on the attribute information, application domain information and application scenario information of the large language model.

[0123] The third generating module 706 is used to perform a baseline behavior detection on the current instruction and generate a third detection result when the second detection result is a passed detection; wherein the third detection result is used to indicate whether the current instruction is a normal behavior.

[0124] The fourth generation module 708 is used to perform instruction role detection on the current instruction when the third detection result is a passed detection, generate a fourth detection result, and interact with the large language model based on the current instruction when the fourth detection result is a passed detection; wherein the fourth detection result is used to indicate whether the current instruction is consistent with the current user role.

[0125] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here. Each module in the above-mentioned large language model instruction risk detection device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations of the above modules.

[0126] In one embodiment of the present application, a computer device is provided, the computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the following steps when executing the computer program:

[0127] Performing a legality check on the current instruction of the large language model to generate a first detection result;

[0128] When the current instruction is determined to be legal based on the first detection result, a domain constraint detection is performed on the current instruction to generate a second detection result; wherein the second detection result is used to indicate whether the current instruction exceeds a corresponding preset constraint condition, and the preset constraint condition is determined based on the attribute information, application domain information, and application scenario information of the large language model;

[0129] When the second detection result is that the detection is passed, a baseline behavior detection is performed on the current instruction to generate a third detection result; wherein the third detection result is used to indicate whether the current instruction is a normal behavior;

[0130] When the third detection result is that the detection is passed, the current instruction is subjected to instruction role detection to generate a fourth detection result, and when the fourth detection result is that the detection is passed, the large language model is interacted based on the current instruction; wherein the fourth detection result is used to indicate whether the current instruction is consistent with the current user role.

[0131] In one embodiment of the present application, when the processor executes the computer program, the processor further implements the following steps:

[0132] A preset pattern matching algorithm is used to perform legality and compliance verification and instruction injection verification on the current instruction to generate a first detection result; wherein, the legality and compliance verification is used to determine whether the current instruction includes illegal content, and the instruction injection verification is used to determine whether the current instruction is other instructions that cover or bypass expected instructions or preset constraints.

[0133] In one embodiment of the present application, when the processor executes the computer program, the processor further implements the following steps:

[0134] Perform type judgment on the current instruction and generate a type judgment result;

[0135] Based on the type judgment result and the preset constraint conditions corresponding to the preset classification category, a second detection result is generated.

[0136] In one embodiment of the present application, when the processor executes the computer program, the processor further implements the following steps:

[0137] Obtaining a preset baseline detection sub-model; wherein the preset baseline detection sub-model is trained based on a sample data set, and the sample data set includes a plurality of historical instruction information, system role setting information, and system interaction information;

[0138] A baseline behavior pattern detection is performed on the current instruction through a preset baseline detection sub-model to generate a third detection result.

[0139] In one embodiment of the present application, when the processor executes the computer program, the processor further implements the following steps:

[0140] Obtain a sample data set and perform data processing on the sample data set to generate a target data set; wherein the data processing includes data preprocessing, standardization processing, word segmentation processing and encoding processing;

[0141] Perform clustering on the target data set, and perform behavioral pattern recognition on the clustering results to obtain recognition results; the recognition results include normal behavior patterns or abnormal behavior patterns, and the normal behavior patterns are used to construct baseline behavior patterns;

[0142] The initial baseline detection sub-model is trained using data corresponding to normal behavior patterns in the target data set to generate a preset baseline detection sub-model.

[0143] In one embodiment of the present application, when the processor executes the computer program, the processor further implements the following steps:

[0144] Perform intention recognition and standardization processing on the current instruction to obtain the processing result;

[0145] Using a preset similarity algorithm to calculate the similarity between the processing result and each instruction in the preset instruction data set;

[0146] Based on the maximum value of the similarity, determine the target instruction level corresponding to the current instruction;

[0147] The target user role corresponding to the target instruction level is compared with the current user role corresponding to the current instruction, and a fourth detection result is generated according to the comparison result.

[0148] In one embodiment of the present application, when the processor executes the computer program, the processor further implements the following steps:

[0149] When any one of the first detection result, the second detection result, the third detection result and the fourth detection result fails the detection, an error message is generated and the interaction process corresponding to the current instruction is terminated.

[0150] The computer device provided in the embodiment of the present application has similar implementation principles and technical effects to those of the above-mentioned method embodiment, and will not be described in detail here.

[0151] In one embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0152] Performing a legality check on the current instruction of the large language model to generate a first detection result;

[0153] When the current instruction is determined to be legal based on the first detection result, a domain constraint detection is performed on the current instruction to generate a second detection result; wherein the second detection result is used to indicate whether the current instruction exceeds a corresponding preset constraint condition, and the preset constraint condition is determined based on the attribute information, application domain information, and application scenario information of the large language model;

[0154] When the second detection result is that the detection is passed, a baseline behavior detection is performed on the current instruction to generate a third detection result; wherein the third detection result is used to indicate whether the current instruction is a normal behavior;

[0155] When the third detection result is that the detection is passed, the current instruction is subjected to instruction role detection to generate a fourth detection result, and when the fourth detection result is that the detection is passed, the large language model is interacted based on the current instruction; wherein the fourth detection result is used to indicate whether the current instruction is consistent with the current user role.

[0156] In one embodiment of the present application, when the computer program is executed by a processor, the following steps are further implemented:

[0157] A preset pattern matching algorithm is used to perform legality and compliance verification and instruction injection verification on the current instruction to generate a first detection result; wherein, the legality and compliance verification is used to determine whether the current instruction includes illegal content, and the instruction injection verification is used to determine whether the current instruction is other instructions that cover or bypass expected instructions or preset constraints.

[0158] In one embodiment of the present application, when the computer program is executed by a processor, the following steps are further implemented:

[0159] Perform type judgment on the current instruction and generate a type judgment result;

[0160] Based on the type judgment result and the preset constraint conditions corresponding to the preset classification category, a second detection result is generated.

[0161] In one embodiment of the present application, when the computer program is executed by a processor, the following steps are further implemented:

[0162] Obtaining a preset baseline detection sub-model; wherein the preset baseline detection sub-model is trained based on a sample data set, and the sample data set includes a plurality of historical instruction information, system role setting information, and system interaction information;

[0163] A baseline behavior pattern detection is performed on the current instruction through a preset baseline detection sub-model to generate a third detection result.

[0164] In one embodiment of the present application, when the computer program is executed by a processor, the following steps are further implemented:

[0165] Obtain a sample data set and perform data processing on the sample data set to generate a target data set; wherein the data processing includes data preprocessing, standardization processing, word segmentation processing and encoding processing;

[0166] Perform clustering on the target data set, and perform behavioral pattern recognition on the clustering results to obtain recognition results; the recognition results include normal behavior patterns or abnormal behavior patterns, and the normal behavior patterns are used to construct baseline behavior patterns;

[0167] The initial baseline detection sub-model is trained using data corresponding to normal behavior patterns in the target data set to generate a preset baseline detection sub-model.

[0168] In one embodiment of the present application, when the computer program is executed by a processor, the following steps are further implemented:

[0169] Perform intention recognition and standardization processing on the current instruction to obtain the processing result;

[0170] Using a preset similarity algorithm to calculate the similarity between the processing result and each instruction in the preset instruction data set;

[0171] Based on the maximum value of the similarity, determine the target instruction level corresponding to the current instruction;

[0172] The target user role corresponding to the target instruction level is compared with the current user role corresponding to the current instruction, and a fourth detection result is generated according to the comparison result.

[0173] In one embodiment of the present application, when the computer program is executed by a processor, the following steps are further implemented:

[0174] When any one of the first detection result, the second detection result, the third detection result and the fourth detection result fails the detection, an error message is generated and the interaction process corresponding to the current instruction is terminated.

[0175] The computer-readable storage medium provided in this embodiment has similar implementation principles and technical effects to those of the above method embodiments, and will not be described in detail here.

[0176] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0177] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0178] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for detecting large language model instruction risks, characterized in that: The method comprises: Performing a legality check on the current instruction of the large language model to generate a first detection result; In the case where the current instruction is determined to be legal based on the first detection result, a domain constraint detection is performed on the current instruction to generate a second detection result; wherein the second detection result is used to indicate whether the current instruction exceeds a corresponding preset constraint condition, and the preset constraint condition is determined based on the attribute information, application domain information, and application scenario information of the large language model; When the second detection result is that the detection is passed, a baseline behavior detection is performed on the current instruction to generate a third detection result; wherein the third detection result is used to indicate whether the current instruction is a normal behavior; When the third detection result is a passed detection, the current instruction is subjected to an instruction role detection to generate a fourth detection result, and when the fourth detection result is a passed detection, an interaction with a large language model is performed based on the current instruction; wherein the fourth detection result is used to indicate whether the current instruction is consistent with the current user role.

2. The method according to claim 1, characterized in that The legality check includes a legality and compliance check and an instruction injection check. The legality check of the current instruction is performed to generate a first detection result, including: A preset pattern matching algorithm is used to perform the legality and compliance check and instruction injection check on the current instruction to generate the first detection result; wherein the legality and compliance check is used to determine whether the current instruction includes illegal content, and the instruction injection check is used to determine whether the current instruction is other instructions that cover or bypass expected instructions or preset constraints.

3. The method according to claim 1 or 2, characterized in that: The performing domain constraint detection on the current instruction to generate a second detection result includes: Performing type judgment on the current instruction and generating a type judgment result; The second detection result is generated based on the type judgment result and the preset constraint conditions corresponding to the preset classification category.

4. The method according to claim 1 or 2, characterized in that: The performing a baseline behavior detection on the current instruction to generate a third detection result includes: Obtaining a preset baseline detection submodel; wherein the preset baseline detection submodel is obtained by training based on a sample data set, and the sample data set includes a plurality of historical instruction information, system role setting information, and system interaction information; The third detection result is generated by performing a baseline behavior pattern detection on the current instruction through the preset baseline detection sub-model.

5. The method according to claim 4, characterized in that The construction process of the preset baseline detection sub-model includes: Acquire a sample data set and perform data processing on the sample data set to generate a target data set; wherein the data processing includes data preprocessing, standardization processing, word segmentation processing and encoding processing; Performing clustering processing on the target data set, and performing behavior pattern recognition on the clustering results to obtain a recognition result; the recognition result includes a normal behavior pattern or an abnormal behavior pattern, and the normal behavior pattern is used to construct a baseline behavior pattern; The initial baseline detection sub-model is trained using data corresponding to the normal behavior pattern in the target data set to generate the preset baseline detection sub-model.

6. The method according to claim 1 or 2, characterized in that: The performing instruction role detection on the current instruction to generate a fourth detection result includes: Performing intention recognition and standardization processing on the current instruction to obtain a processing result; Calculate the similarity between the processing result and each instruction in the preset instruction data set using a preset similarity algorithm; Based on the maximum value of the similarities, determining a target instruction level corresponding to the current instruction; The target user role corresponding to the target instruction level is compared with the current user role corresponding to the current instruction, and the fourth detection result is generated according to the comparison result.

7. The method according to claim 1 or 2, characterized in that: The method further comprises: When any one of the first detection result, the second detection result, the third detection result and the fourth detection result fails the detection, an error message is generated and the interaction process corresponding to the current instruction is terminated.

8. A large language model instruction risk detection device, characterized in that: The device comprises: A first generating module, used to perform a legality check on the current instruction of the large language model and generate a first detection result; A second generation module is used to perform a domain constraint detection on the current instruction to generate a second detection result when the current instruction is determined to be legal based on the first detection result; wherein the second detection result is used to indicate whether the current instruction exceeds a corresponding preset constraint condition, and the preset constraint condition is determined based on the attribute information, application domain information, and application scenario information of the large language model; A third generating module, configured to perform a baseline behavior detection on the current instruction to generate a third detection result when the second detection result is a detection pass; wherein the third detection result is used to indicate whether the current instruction is a normal behavior; The fourth generation module is used to perform instruction role detection on the current instruction when the third detection result is a passed detection, generate a fourth detection result, and interact with the large language model based on the current instruction when the fourth detection result is a passed detection; wherein the fourth detection result is used to indicate whether the current instruction is consistent with the current user role.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method for detecting large language model instruction risks as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the large language model instruction risk detection method as described in any one of claims 1-7.

Images