Honey point-based big language model prompt injection detection method
By constructing and deploying prompt word honey dots in large language models, efficient detection and defense of prompt injection attacks is achieved, and the problems of high defense costs and low detection success rate in the existing technology are solved, improving detection efficiency and adaptability .
Patent Information
- Application Number
- CN202411120043.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-15
- Publication Date
- 2024-11-12
AI Technical Summary
When facing prompt injection attacks, existing large language model content security technology has high defense costs, low detection success rate, large time overhead, and difficult update and iteration, making it difficult to effectively prevent content that does not comply with laws, regulations or ethical norms.
Using a large language model prompt injection detection method based on honey dots, the prompt injection instruction is used to detect text to be detected based on content security regulations, and the text to be detected is detected based on content security regulations. Real-time update prompts to inject the library to deal with update attacks.
It reduces defense costs, improves detection success rate, reduces time overhead, and can effectively respond to update prompt injection attacks to adapt to real-time update requirements.
Smart Images

Figure CN118940756A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of network security technology, and in particular to the field of large language model content security technology. Background Art
[0002] The large language model is an algorithm based on deep neural networks. Its main task is to learn statistical laws in massive text data, so as to achieve tasks such as generation, classification, and translation of given input text. Prompt injection refers to an attack in which an attacker constructs specific prompt words to bypass the limitations of the large language model and destroy the model's security alignment, thereby generating an attack form that does not comply with laws, regulations, or ethical norms.
[0003] Existing methods build model firewalls and use content detection models to conduct risk assessment and harmful content filtering on content generated by large language models. However, these security enhancement methods have problems such as high training defense costs, low detection success rate, high time overhead, and difficulty in updating and iteration when facing prompt injection attacks. Therefore, it is urgent to provide a solution to improve the above problems. Summary of the invention
[0004] The purpose of the present invention is to provide a large language model prompt injection detection method based on honey spots, which improves the problems of high defense cost, low detection success rate, large time overhead and difficult update iteration in the prior art.
[0005] The present invention provides a honey spot-based large language model prompt injection detection method, which adopts the following technical solutions:
[0006] Building a prompt word honey spot based on the content security regulations of the large language model and the prompt injection library, and deploying the prompt word honey spot in the system prompt word library of the large language model to be protected to obtain a protected model;
[0007] Acquire texts to be detected for multiple user instructions based on the protected model;
[0008] When the prompt word honey spot is triggered, the protected model determines whether the plurality of user instructions are prompt injection instructions, intercepts the instructions when the user instructions are prompt injection instructions, detects the text to be detected based on the content security regulations, and outputs the detection result.
[0009] The honey spot-based large language model prompt injection detection method, wherein the process of constructing the prompt word honey spot based on the content security provisions of the large language model and the prompt injection library includes:
[0010] The content security regulations and the prompt injection library of the large language model are obtained based on the literature, and the contextual semantic understanding and association analysis of the content security regulations and the prompt injection library are performed based on the large language model to construct prompt word honey spots.
[0011] As described above, in the method for detecting large language model prompt injection based on honey spots, in the process of deploying the prompt word honey spots in the system prompt word library of the large language model to be protected, the prompt word honey spots are deployed in the system prompt word library of the model to be protected in the form of prompt words.
[0012] The honey spot-based large language model prompt injection detection method as described above, wherein obtaining the to-be-detected text for multiple user instructions based on the protected model comprises:
[0013] A user prompt word is input into the protected model, the user prompt word includes a plurality of user instructions, and the protected model generates a text to be detected based on the plurality of user instructions.
[0014] As described above, the large language model prompt injection detection method based on honey spots, wherein, in the process of user prompt words inputting into the protected model, the protected model includes: before generating the text to be detected, the protected model will add the system prompt words in the context of the user prompt words to trigger the prompt word honey spots.
[0015] The honey spot-based large language model prompt injection detection method as described above, wherein the prompt word honey spot is triggered including:
[0016] When there is a prompt injection instruction in the multiple user instructions and it is not recognized by the protected model, the prompt word honey spot is not triggered, and the protected model directly outputs the text to be detected, otherwise the prompt word honey spot is triggered.
[0017] As described above, the large language model prompt injection detection method based on honey spots, wherein the process in which the protected model performs prompt injection judgment on the multiple user instructions includes: the protected model performs semantic analysis on the multiple user instructions, and judges whether the multiple user instructions are prompt injection instructions based on the prompt injection library.
[0018] The honey spot-based large language model prompt injection detection method as described above, wherein the process of detecting the text to be detected based on the content security provision includes:
[0019] The protected model performs semantic analysis on the text to be detected to determine whether the text to be detected complies with the content security regulations. When it complies with the content security regulations, the protected model outputs the text; otherwise, the text to be detected is marked, the protected model outputs a rejection answer message, and prompts the injection library to be updated based on the mark.
[0020] The honey spot-based large language model prompt injection detection method as described above, wherein the process of updating the prompt injection library based on the tag includes:
[0021] Based on the understanding and analysis of the protected model, the similarity between the prompt injection instruction and the known prompt injection words in the prompt injection library is obtained. When the similarity is higher than a threshold, it is judged as a known attack. Otherwise, the prompt injection library is updated, and the prompt word honey spot is updated.
[0022] The large language model prompt injection detection method based on honey spots as described above, wherein the process of updating the prompt word honey spots includes: real-time updating of documents related to content security based on the large language model, and real-time updating of the prompt injection library, thereby updating the prompt word honey spots.
[0023] The beneficial effects of the honey spot-based large language model prompt injection detection method proposed in the present invention are:
[0024] 1. The prompt word honey spot of the present invention is deployed in the protected model in the form of system prompt words, with low deployment cost and no need to introduce additional training models.
[0025] 2. The protected model of the present invention directly generates the text to be detected and uses the protected model itself and the prompt word honey spots for detection, thereby improving the detection success rate and reducing time expenditure.
[0026] 3. The prompt word honey spot of the present invention can be updated in real time based on the prompt injection instruction to adapt to the newer prompt injection attack. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 A flow chart of a honey spot-based large language model prompt injection detection method provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0028] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Unless otherwise defined, the technical terms or scientific terms used herein should be understood by people with general skills in the field to which the present invention belongs. "Including" and similar words used in this article mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects.
[0029] The embodiment of the present invention provides a large language model prompt injection detection method based on honey spots, see Figure 1 ,include:
[0030] S1. Construct a prompt word honey spot based on the content security regulations of the large language model and the prompt injection library, and deploy the prompt word honey spot in the system prompt word library of the large language model to be protected to obtain a protected model;
[0031] S2. Acquire texts to be detected for multiple user instructions based on the protected model;
[0032] S3. When the prompt word honey spot is triggered, the protected model determines whether the plurality of user instructions are prompt injection instructions, intercepts the instructions when the user instructions are prompt injection instructions, detects the text to be detected based on the content security regulations, and outputs the detection result.
[0033] In some embodiments, the process of executing step S1 includes:
[0034] S1.1. Construct prompt word honey spots based on content security regulations and prompt injection library of large language model;
[0035] S1.2. Deploy the prompt word honey spot in the system prompt word library of the large language model to be protected.
[0036] Specifically, when executing step S1.1, constructing a prompt word honey spot based on the content security provisions of the large language model and the prompt injection library includes:
[0037] Based on the literature, the content security regulations and the prompt injection library of the large language model are obtained, and the contextual semantic understanding and association analysis of the content security regulations and the prompt injection library are performed based on the large language model to construct prompt word honey spots.
[0038] In fact, the prompt injection instructions in the prompt injection library can obtain the permissions of non-normal users of the system, causing the security measures to fail.
[0039] Furthermore, the prompt injection library can be updated in real time according to the prompt injection sample acquisition in the network and the detected prompt injection attack.
[0040] Specifically, when executing step S1.2, deploying the prompt word honey spot in the system prompt word library of the large language model to be protected includes:
[0041] The prompt word honey spot is deployed in the system prompt word library of the model to be protected in the form of prompt words.
[0042] In some embodiments, in the process of executing step S2, obtaining the text to be detected for multiple user instructions based on the protected model includes:
[0043] A user prompt word is input into the protected model, the user prompt word includes a plurality of user instructions, and the protected model generates a text to be detected based on the plurality of user instructions.
[0044] Specifically, during the process of inputting the user prompt word into the protected model, the protected model will add the system prompt word to the context of the user prompt word before generating the text to be detected to trigger the prompt word honey spot.
[0045] In some embodiments, the process of executing step S3 includes:
[0046] S3.1, the prompt word honey spot is triggered;
[0047] S3.2, the protected model judges the prompt injection command for the plurality of user commands, and intercepts the command when the user command is a prompt injection command;
[0048] S3.3. Detect the text to be detected based on the content security regulations and output the detection result.
[0049] Specifically, when executing step S3.1, the prompt word honey spot is triggered, if there is a prompt injection instruction in the multiple user instructions and it is not recognized by the protected model, the prompt word honey spot is not triggered, and the protected model directly outputs the text to be detected, otherwise the prompt word honey spot is triggered.
[0050] In fact, the prompt injection attack method is artificially studied and constantly updated, so there is no guarantee that all prompt injection instructions input into the protected model can be intercepted. At this time, the prompt injection library can be updated based on the abnormal phenomenon that the prompt word honey spot is not triggered but the text to be detected is output.
[0051] Specifically, when executing step S3.2, the protected model performs prompt injection instruction judgment on the plurality of user instructions, including:
[0052] The protected model performs semantic analysis on the plurality of user instructions, and determines whether the plurality of user instructions are prompt injection instructions based on the prompt injection library.
[0053] Furthermore, when a prompt injection instruction exists among the plurality of user instructions, the prompt word honey spot intercepts the prompt injection instruction, so that subsequent prompt word honey spots can detect the text to be detected.
[0054] Specifically, when executing step S3.3, detecting the text to be detected based on the content security provision and outputting the detection result, it includes:
[0055] The protected model performs semantic analysis on the text to be detected to determine whether the text to be detected complies with the content security regulations. When it complies with the content security regulations, the protected model outputs the text; otherwise, the text to be detected is marked, the protected model outputs a rejection answer message, and prompts the injection library to be updated based on the mark.
[0056] Among them, the text to be detected does not comply with the content security regulations but is not detected and is output by the protected model. At this time, it is the probability that the prompt word honey spot is not successfully detected.
[0057] Furthermore, based on the understanding and analysis of the protected model, the similarity between the prompt injection instruction and the known prompt injection words in the prompt injection library is obtained. When the similarity is higher than a threshold, it is judged as a known attack, otherwise the prompt injection library is updated, and the prompt word honey spot is updated.
[0058] Furthermore, based on the real-time update of the content security-related literature of the large language model and the real-time update of the prompt injection library, the prompt word honey spots are updated.
[0059] In fact, the sum of the probabilities that the prompt word honey spot is not triggered and that the text to be detected does not comply with the content security regulations but is not detected is less than 10%, that is, the probability of successful detection is greater than 90%.
[0060] Although the embodiments of the present invention are described in detail above, it is obvious to those skilled in the art that various modifications and variations can be made to these embodiments. However, it should be understood that such modifications and variations are within the scope and spirit of the present invention as described in the claims. Moreover, the present invention described herein may have other embodiments and may be implemented or realized in a variety of ways.
Claims
1. A honey spot-based large language model hint injection detection method, It is characterized in that The following steps are involved: Building a prompt word honey spot based on the content security regulations of the large language model and the prompt injection library, and deploying the prompt word honey spot in the system prompt word library of the large language model to be protected to obtain a protected model; Acquire texts to be detected for multiple user instructions based on the protected model; When the prompt word honey spot is triggered, the protected model determines whether the plurality of user instructions are prompt injection instructions, intercepts the instructions when the user instructions are prompt injection instructions, detects the text to be detected based on the content security regulations, and outputs the detection result.
2. According to the honey spot-based large language model hint injection detection method of claim 1, It is characterized in that The process of building a hint word honey spot based on the content security regulations and hint injection library of the large language model includes: The content security regulations and the prompt injection library of the large language model are obtained based on the literature, and the contextual semantic understanding and association analysis of the content security regulations and the prompt injection library are performed based on the large language model to construct prompt word honey spots.
3. The honey spot-based large language model hint injection detection method according to claim 1, It is characterized in that The process of deploying the prompt word honey spot in the system prompt word library of the large language model to be protected includes: deploying the prompt word honey spot in the system prompt word library of the model to be protected in the form of prompt words.
4. The honey spot-based large language model hint injection detection method according to claim 1, It is characterized in that Acquiring the text to be detected for multiple user instructions based on the protected model includes: A user prompt word is input into the protected model, the user prompt word includes a plurality of user instructions, and the protected model generates a text to be detected based on the plurality of user instructions.
5. The honey spot-based large language model hint injection detection method according to claim 4, It is characterized in that The process of the user prompting words to enter the protected model includes: Before generating the text to be detected, the protected model adds a system prompt word to the user prompt word context to trigger the prompt word honey spot.
6. The honey spot-based large language model hint injection detection method according to claim 5, It is characterized in that The prompt word honey spot is triggered including: When there is a prompt injection instruction in the multiple user instructions and it is not recognized by the protected model, the prompt word honey spot is not triggered, and the protected model directly outputs the text to be detected, otherwise the prompt word honey spot is triggered.
7. The honey spot-based large language model hint injection detection method according to claim 1, It is characterized in that The protected model performs prompt injection judgment on the plurality of user instructions, including: The protected model performs semantic analysis on the plurality of user instructions, and determines whether the plurality of user instructions are prompt injection instructions based on the prompt injection library.
8. The honey spot-based large language model hint injection detection method according to claim 1, It is characterized in that The process of detecting the text to be detected based on the content security provision includes: The protected model performs semantic analysis on the text to be detected to determine whether the text to be detected complies with the content security regulations. When it complies with the content security regulations, the protected model outputs the text; otherwise, the text to be detected is marked, the protected model outputs a rejection answer message, and prompts the injection library to be updated based on the mark.
9. The honey spot-based large language model hint injection detection method according to claim 8, It is characterized in that The process of updating the prompt injection library based on the tag includes: Based on the understanding and analysis of the protected model, the similarity between the prompt injection instruction and the known prompt injection words in the prompt injection library is obtained. When the similarity is higher than a threshold, it is judged as a known attack. Otherwise, the prompt injection library is updated, and the prompt word honey spot is updated.
10. The honey spot-based large language model hint injection detection method according to claim 9, It is characterized in that The process of updating the prompt word honey spot includes: Based on the real-time update of the content security-related literature of the large language model and the real-time update of the prompt injection library, the prompt word honey spots are updated.
