Method, model and equipment for identifying hint injection attack aiming at large language model

Abstract

The invention relates to the field of artificial intelligence and natural language processing, in particular to a method, a model and equipment for identifying a prompt injection attack for a large language model. The method comprises the following steps: acquiring a cue word to be input into an LLM; performing semantic analysis on the cue word to obtain a semantic analysis result; performing prompt injection word detection on the prompt word to obtain a prompt injection word detection result; and according to a semantic analysis result and a prompt injection word detection result, analyzing whether the prompt word relates to a prompt injection attack. The invention provides a fusion recognition mechanism combining semantic analysis and prompt injection word detection, and the fusion recognition mechanism is specially used for recognizing the prompt injection attack aiming at the LLM, so that the defense capability for the LLM prompt injection attack is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence and natural language processing, and in particular, to a method, model, and device for identifying prompt injection attacks against large language models (LLMs). Background Art

[0002] With the widespread application of LLMs, their powerful capabilities in processing text have also brought new security challenges. When processing user input, LLMs have difficulty distinguishing between system instructions and user data, and this ambiguity of the boundary makes prompt injection attacks possible. Attackers induce the model to generate harmful content or leak sensitive information by embedding malicious instructions in the input, seriously threatening the security of LLM application systems.

[0003] In related technologies, the defense methods against LLM prompt injection attacks are still in the initial exploration stage. The current content security technologies are mainly used to identify violations and bad information in web pages, but these technologies are mainly based on known harmful content patterns and are difficult to cope with the complexity and concealment of prompt injection attacks. No reasonable and effective method for identifying prompt injection attacks against LLMs has been provided in related technologies. Summary of the Invention

[0004] In view of this, the present disclosure proposes a method, model, and device for identifying prompt injection attacks against LLMs, which adopts a fusion recognition mechanism combining semantic analysis and prompt injection word detection, and can be specifically used to identify prompt injection attacks against LLMs, thereby enhancing the defense ability against LLM prompt injection attacks.

[0005] According to one aspect of the present disclosure, a method for identifying prompt injection attacks against LLMs is provided. The method includes:

[0006] Obtain a prompt word to be input into the LLM;

[0007] Perform semantic analysis on the prompt word to obtain a semantic analysis result;

[0008] Perform prompt injection word detection on the prompt word to obtain a prompt injection word detection result;

[0009] Analyze whether the prompt word involves a prompt injection attack according to the semantic analysis result and the prompt injection word detection result.

[0010] In a possible implementation, the performing semantic analysis on the prompt word includes:

[0011] Use a trained neural network model to perform context-based semantic analysis on the prompt word.

[0012] In another possible implementation, the detecting the prompt injection words for the prompt words includes:

[0013] Detecting whether the prompt words include keywords that match the prompt injection words in a pre-constructed prompt injection word library, or

[0014] Converting the prompt words into a structured query statement, and retrieving whether the prompt words include prompt injection words in a knowledge base or a knowledge graph according to the query statement.

[0015] In another possible implementation, the semantic analysis result includes a probability value that the prompt words have a normal semantic expression not involving prompt injection attacks, and

[0016] Analyzing whether the prompt words involve prompt injection attacks according to the semantic analysis result and the prompt injection word detection result, including:

[0017] When the probability value is greater than or equal to a first threshold, determining that the prompt words do not involve prompt injection attacks,

[0018] When the probability value is less than or equal to a second threshold, determining that the prompt words involve prompt injection attacks, where the second threshold is less than the first threshold, and

[0019] When the probability value is less than the first threshold and greater than the second threshold, further analyzing whether the prompt words involve prompt injection attacks according to the prompt injection word detection result.

[0020] In another possible implementation, further analyzing whether the prompt words involve prompt injection attacks according to the prompt injection word detection result, including:

[0021] When the prompt words do not include prompt injection words, determining that the prompt words do not involve prompt injection attacks, and

[0022] When the prompt words include prompt injection words, determining that the prompt words involve prompt injection attacks.

[0023] In another possible implementation, the method further includes:

[0024] When the prompt words do not involve prompt injection attacks, allowing the prompt words to be input into the LLM;

[0025] When the prompt words involve prompt injection attacks, preventing the prompt words from being input into the LLM, and outputting a prompt message indicating that the prompt words involve prompt injection attacks.

[0026] According to another aspect of the present disclosure, there is provided a fusion recognition model for identifying prompt injection attacks against an LLM, the fusion recognition model comprising:

[0027] A neural network model for performing semantic analysis on the prompts to be input into the LLM to obtain a semantic analysis result;

[0028] A prompt injection word detection module for detecting prompt injection words in the prompts to obtain a prompt injection word detection result;

[0029] A model fusion module for analyzing whether the prompts involve prompt injection attacks according to the semantic analysis result and the prompt injection word detection result.

[0030] According to another aspect of the present disclosure, there is provided a computing device comprising a memory, a processor, and a computer program stored on the memory, the processor executing the computer program to implement the operations of the above method.

[0031] According to another aspect of the present disclosure, there is provided a non-volatile computer-readable storage medium having a computer program stored thereon, the computer program, when executed by a processor, implementing the operations of the above method.

[0032] According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program, or a non-volatile computer-readable storage medium carrying the computer program, the computer program, when executed by a processor, implementing the operations of the above method.

[0033] An embodiment of the present disclosure proposes a method for identifying prompt injection attacks against an LLM. By comprehensively considering the semantic analysis result and the prompt injection word detection result, this method comprehensively and accurately analyzes the prompts to be input into the LLM, so as to efficiently identify whether the prompts involve prompt injection attacks. This fusion recognition architecture not only significantly improves the recognition accuracy of prompt injection attacks, but also greatly reduces the risks of misjudgment and missed judgment. Through strict semantic analysis and prompt word detection mechanisms, this method constructs a solid and efficient defense barrier for the LLM, greatly enhancing the system's ability to resist prompt injection attacks and providing strong guarantee for the secure operation of the LLM.

[0034] According to the following detailed description of exemplary embodiments with reference to the accompanying drawings, other features and aspects of the present disclosure will become apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings, which are included in and constitute a part of this specification, illustrate exemplary embodiments, features, and aspects of the present disclosure and are used to explain the principles of the present disclosure.

[0036] Figure 1 The flowchart of the method for identifying prompt injection attacks against LLMs provided by an exemplary embodiment of the present disclosure is shown.

[0037] Figure 2 The schematic structural diagram of the fusion recognition model provided by an exemplary embodiment of the present disclosure is shown.

[0038] Figure 3 The flowchart of the method for identifying prompt injection attacks against LLMs provided by another exemplary embodiment of the present disclosure is shown.

[0039] Figure 4 The flowchart of the model fusion algorithm of the model fusion module provided by an exemplary embodiment of the present disclosure is shown.

[0040] Figure 5 The block diagram of a device shown according to an exemplary embodiment. Detailed implementation manners

[0041] The various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the drawings denote elements with the same or similar functions. Although various aspects of the embodiments are shown in the drawings, the drawings do not have to be drawn to scale unless otherwise specified.

[0042] As used herein, the terms "comprising," "including," "having," or their variants are open-ended and include one or more stated features, wholes, elements, steps, components, or functions, but do not exclude the existence or addition of one or more other features, wholes, elements, steps, components, functions, or groups thereof.

[0043] When an element is referred to as being "connected," "coupled," "responsive," or a variant thereof to another element, it can be directly connected, coupled, or responsive to the other element, or intervening elements may be present.

[0044] Although the terms first, second, third, etc. may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Thus, without departing from the teachings of the inventive concept, the first element / operation in some embodiments may be referred to as the second element / operation in other embodiments.

[0045] The term "exemplary" as used herein means "serving as an example, embodiment, or illustration." Any embodiment described as "exemplary" herein need not be construed as superior to or better than other embodiments.

[0046] In addition, for a better illustration of the present disclosure, numerous specific details are provided in the following detailed implementation manners. Those skilled in the art should understand that the present disclosure can still be implemented without some specific details. In some instances, methods, means, elements, and circuits well-known to those skilled in the art are not described in detail to highlight the gist of the present disclosure.

[0047] Prompt injection attack is a security threat against LLMs. Attackers insert specific instructions or logic into the prompts to induce the model to generate incorrect or harmful content. The behavior of prompt injection attacks may be used to bypass the security mechanisms of the model, obtain sensitive information, or spread false content. The characteristics of prompt injection attacks against LLMs are as follows: 1. Insert specific statements into the prompts, such as "ignore above instructions"; 2. Abrupt semantic transitions in the context. In view of the above characteristics, the embodiments of the present disclosure propose a fusion recognition mechanism combining semantic analysis and prompt injection word detection, which can be specifically used to identify prompt injection attacks against LLMs, thereby enhancing the defense ability against prompt injection attacks on LLMs.

[0048] The method for identifying prompt injection attacks against LLMs proposed by the embodiments of the present disclosure can be widely applied to various computing devices. The computing device can be a terminal or a server. The terminal includes a mobile terminal or a fixed terminal. For example, the terminal can be a mobile phone, a tablet computer, a laptop computer, a desktop computer, etc. The server can be a single server, or a server cluster composed of several servers, or a cloud computing service center. In the computing device, a fusion recognition model is deployed, which combines semantic analysis and prompt injection word detection and is specifically used to identify prompt injection attacks against LLMs.

[0049] Next, several exemplary embodiments are used to introduce the method for identifying prompt injection attacks against LLMs provided by the embodiments of the present disclosure.

[0050] Please refer to Figure 1 , which shows a flowchart of the method for identifying prompt injection attacks against LLMs provided by an exemplary embodiment of the present disclosure. In this embodiment, it is exemplified that the computing device executes this method. The method includes the following steps.

[0051] Step 101, obtain the prompt to be input into the LLM.

[0052] LLM is an artificial intelligence model built based on deep learning technology, usually with a parameter scale of billions or even hundreds of billions. Through unsupervised learning on a vast amount of text data, it can learn the grammar, semantics, and context relationships of language and generate natural language text. LLM can be used to understand and generate human language and is widely applied in fields such as text summarization, translation, sentiment analysis, and dialogue systems. For example, in the field of text summarization, LLM is used to automatically extract or generate the core content of the text to help users quickly obtain key information. In the field of translation, LLM can perform high-quality text translation, converting the text of one language into the text of another language while maintaining semantic accuracy and fluency. In the field of sentiment analysis, LLM is used to identify and analyze the sentiment tendency in the text and is widely applied in market analysis, etc. In the field of dialogue systems, LLM is used to construct dialogue texts that can communicate naturally with human users and is used in occasions such as customer service and entertainment. The application scenarios of the LLM in the embodiments of the present disclosure are not limited.

[0053] A prompt is the text content of natural language that a user needs to input into the LLM, used to guide the LLM to generate corresponding answers or perform specific tasks. The prompt is the starting point of the interaction between the user and the LLM and determines how the LLM understands and responds to the user's needs. The prompt can be a question, instruction, or description that the user presents to the LLM through a user input interface (such as a chat box, text input box, etc.), and can also include context information or constraints. The LLM will generate relevant answers based on the content of the prompt. For example, the user inputs: "Please tell me the definition of artificial intelligence." "Please tell me the definition of artificial intelligence." is the prompt, which guides the LLM to generate an explanation of the definition of artificial intelligence.

[0054] A computing device for executing a method for identifying a prompt injection attack against an LLM can, for example, collect the prompts to be input into the LLM through a prompt collection module (not shown). The prompt collection module can, for example, be an open interface, enabling the computing device to efficiently and real-time receive prompts from different sources, so as to be able to simultaneously identify prompt injection attacks against one or more LLMs.

[0055] After obtaining the prompt, the computing device can perform preliminary preprocessing on the prompt, aiming to convert the original natural language text into a format suitable for subsequent model analysis, providing a clearer and more accurate input for subsequent prompt injection word detection or semantic analysis. The preprocessing can include, but is not limited to, word segmentation, stop word removal, non-Chinese information removal, text cleaning, standardization, etc. After the above preprocessing, the output text data will be more standardized, concise, and easier to be processed by machines. Subsequently, the computing device can perform word vector representation on these text data, converting them into a numerical form that can be understood and processed by the neural network model. This process not only improves the operability of the text data but also lays a solid foundation for subsequent tasks such as semantic analysis and prompt injection word detection.

[0056] The purpose of this step is to extract the user's input content for further analysis and processing later.

[0057] Step 102: Perform semantic analysis on the prompt to obtain the semantic analysis result.

[0058] Use natural language processing techniques (such as a trained neural network model) to perform semantic analysis on the prompt to obtain the semantic analysis result. Semantic analysis is an important task in natural language processing, aiming to understand the meaning, intention, and context relationship of the text. Semantic analysis can be implemented based on a neural network, for example, which can capture the semantic information in the text and be used to judge the intention, sentiment, or theme of the text.

[0059] The semantic analysis result refers to the output result obtained from the semantic analysis of the prompt. The semantic analysis result can be: the prompt has an abnormal semantic expression related to a prompt injection attack, the prompt has a normal semantic expression not related to a prompt injection attack, or the probability value of the prompt having a normal semantic expression not related to a prompt injection attack.

[0060] Step 103: Perform prompt injection word detection on the prompt to obtain the prompt injection word detection result.

[0061] Prompt injection word detection includes detecting whether the prompt contains a preset prompt injection word. A prompt injection word is also called a sensitive word, which refers to a specific word or phrase maliciously injected into the prompt and may be used by malicious users to manipulate or attack the language model. These words or phrases usually have a specific pattern or intention and can induce the model to generate unexpected outputs.

[0062] The prompt injection word detection result can be: the prompt does not include a preset prompt injection word, or the prompt includes a preset prompt injection word.

[0063] It should be noted that step 102 and step 103 can be executed in parallel, or step 102 can be executed first and then step 103, or step 103 can be executed first and then step 102. The embodiments of the present disclosure do not limit this.

[0064] Step 104, according to the semantic analysis result and the prompt injection word detection result, analyze whether the prompt word involves a prompt injection attack.

[0065] Combining the semantic analysis result and the prompt injection word detection result, comprehensively analyze whether the prompt word involves a prompt injection attack. According to the final analysis result, low-risk prompt words can be allowed to be input into the LLM, and high-risk prompt words can be blocked or warned. The final analysis result can be output to the user through the recognition result output module (not shown) of the computing device, or a prompt message can be output to the user through the recognition result output module when the prompt word involves a prompt injection attack.

[0066] The specific implementation details of the above steps 102 to 104 will be described in detail later with reference to Figure 3 Detailed description.

[0067] In summary, the embodiments of the present disclosure propose a method for identifying prompt injection attacks against the LLM. By comprehensively considering the semantic analysis result and the prompt injection word detection result, this method comprehensively and accurately analyzes the prompt words to be input into the LLM, so as to efficiently identify whether the prompt words involve prompt injection attacks. This fusion recognition architecture not only significantly improves the recognition accuracy of prompt injection attacks, but also greatly reduces the risks of misjudgment and missed judgment. Through a strict semantic analysis and prompt word detection mechanism, this method constructs a solid and efficient defense barrier for the LLM, greatly enhancing the system's ability to resist prompt injection attacks and providing strong guarantee for the safe operation of the LLM.

[0068] In a possible implementation, the computing device is equipped with a pre-trained fusion recognition model, obtains the prompt words to be input into the LLM, and the fusion recognition model comprehensively analyzes the prompt words through semantic analysis and prompt injection word detection based on a neural network to obtain the final recognition result of whether the prompt words involve prompt injection attacks. The fusion recognition model can be trained through the training methods in the prior art based on a large number of prompt word samples and corresponding recognition result labels.

[0069] Please refer to Figure 2, which shows a schematic structural diagram of a fusion recognition model provided by an exemplary embodiment of the present disclosure. The fusion recognition model 20 is used to analyze whether a prompt involves a prompt injection attack. It is a system architecture that combines semantic analysis, prompt injection word detection, and fusion technology, aiming to comprehensively evaluate the prompts to be input into the LLM and identify prompt injection attacks through multi-dimensional analysis and decision fusion. It consists of multiple independent modules, each module is responsible for processing specific tasks or performing specific analyses, and finally, a fusion module fuses the analysis results of multiple modules to output the final recognition result.

[0070] As Figure 2 shown, the fusion recognition model 20 may include, for example: a neural network model 21, a prompt injection word detection module 22, and a model fusion module 23.

[0071] The neural network model 21 may be a computational model constructed based on artificial neurons and connection weights. This computational model can simulate the information processing method of biological neural networks. By learning a large number of input-output data pairs, it can automatically extract features and patterns in the data, thereby achieving the approximation and prediction of complex functions. The neural network model may select a model based on a deep learning architecture. In this embodiment, the neural network model may adopt a Long Short-Term Memory (LSTM) network. LSTM is a special Recurrent Neural Network (RNN). Through a unique gating mechanism, it can effectively capture long-term dependencies in sequential data and accurately understand and identify context information. This characteristic makes LSTM particularly suitable for analyzing semantic features based on context when processing natural language text. It should be understood that other neural networks may also be adopted for the neural network model as long as they can implement the functions in the present disclosure.

[0072] In the embodiment of the present disclosure, the neural network model 21 may be a pre-trained (learned) LSTM neural network model pre-trained based on a large number of prompt samples and corresponding semantic analysis result labels through the training method in the prior art. When the prompt to be input into the LLM is input into the neural network model 21, the neural network model 21 performs context-based semantic analysis on the prompt to obtain a semantic analysis result. The semantic analysis result may include whether the prompt has a normal semantic expression that does not involve a prompt injection attack and / or the probability value that the prompt has a normal semantic expression that does not involve a prompt injection attack. The semantic analysis result output by the neural network model 21 and its corresponding prompt may be used to further train the neural network model 21 to help the model continuously optimize and adjust model parameters to improve the accuracy of semantic analysis of the prompt.

[0073] The prompt injection word detection module 22 can be a specially designed software component or algorithm module for detecting whether the input data (prompt word) contains preset prompt injection words. The prompt injection word detection module can detect whether the prompt word to be input into the LLM includes prompt injection words through keyword matching, data query, or other detection techniques, using a prompt injection word library or a knowledge base / knowledge graph. The output result of this module is used to assist in judging the security of the prompt word to be input into the LLM.

[0074] In the prompt injection word detection module, a prompt injection word library can be pre-constructed based on previously obtained and / or artificially designed specific words or phrases that may be used by malicious users to manipulate or attack the LLM. The words or phrases in the prompt injection word library are analyzed and sorted to help the prompt injection word detection module quickly identify whether the prompt word contains potential malicious words. The prompt injection word library can support dynamic maintenance, including functions such as adding, deleting, updating, backing up, and rolling back prompt injection words. This enables the prompt injection word library to be adjusted in a timely manner according to the latest prompt injection attack patterns, ensuring the effectiveness of the prompt injection word detection module.

[0075] In addition, the prompt injection word detection module can be connected to a knowledge base or knowledge graph constructed based on previously obtained and / or artificially designed prompt injection patterns. The knowledge base or knowledge graph is a structured data storage form used to store various known prompt injection patterns. These prompt injection patterns are analyzed and sorted to identify potential malicious words or phrases. The stored content of the knowledge base or knowledge graph can include not only prompt injection words but also the context information, semantic associations, and combination patterns of these prompt injection words, etc. These information together constitute the "known prompt injection patterns". By querying data in the knowledge base / knowledge graph, it can be detected whether the prompt word is associated with the known prompt injection patterns, thereby determining whether the prompt word contains preset prompt injection words.

[0076] The model fusion module 23 is a software or algorithm module for comprehensively processing the output results of multiple independent analysis modules (such as the neural network model 21 and the prompt injection word detection module 22). Its core function is to analyze whether the prompt word to be input into the LLM involves a prompt injection attack according to preset logical rules, combining the analysis results of different modules.

[0077] Based on the structure and function of the above fusion recognition model, as Figure 3 shown, the above steps 102 to 104 can be specifically implemented as the following steps.

[0078] Step 301, using the trained neural network model, perform context-based semantic analysis on the prompt word to be input into the LLM to obtain the semantic analysis result.

[0079] A computing device for identifying prompt injection attacks against an LLM can, for example, run the integrated recognition model 20 installed on it and use the neural network model 21 to analyze the prompts to be input into the LLM. This analysis process can mainly perform semantic analysis on the prompts based on the context, and analyze the rationality of the prompts in terms of language logic and semantic expression.

[0080] To enable the prompts to be processed by the neural network model, the prompts can be preprocessed first, including operations such as word segmentation and word vectorization. The preprocessed prompts are input into the trained neural network model, and the neural network model performs context-based semantic analysis on them and outputs the semantic analysis results. The output of the neural network model can be, for example, a probability value indicating the probability that the prompt has a normal semantic expression not involving a prompt injection attack. For example, a probability value of 0.9 means that the prompt is very likely to be normal, that is, it has a normal semantic expression not involving a prompt injection attack, while a probability value of 0.1 means that the prompt may have a problem, that is, it has an abnormal semantic expression involving a prompt injection attack. In some embodiments, this probability value can be directly used as the semantic analysis result. For example, only a numerical value such as "0.9" can be output, or a text such as "The probability that the prompt does not involve a prompt injection attack is 0.9" can be output. Or, in some embodiments, a binarization threshold can be set to binarize the probability value obtained by the neural network model. For example, the binarization threshold can be set to 0.6, and the probability value greater than 0.6 is set to 1 to indicate that the prompt has a normal semantic expression not involving a prompt injection attack, and the probability value less than or equal to 0.6 is set to 0 to indicate that the prompt has an abnormal semantic expression involving a prompt injection attack. In this case, for example, only numerical values such as "0" or "1" can be output, or texts such as "The prompt involves a prompt injection attack" or "The prompt does not involve a prompt injection attack" can be output.

[0081] Step 302, detect the prompt for prompt injection words through the prompt injection word detection module to obtain the prompt injection word detection result.

[0082] In some embodiments, a computing device running the fusion recognition model 20 utilizes the prompt injection word detection module 22 of the fusion recognition model 20 to detect whether a prompt word includes a prompt injection word. In some embodiments, for example, through keyword matching technology, it can be detected whether the prompt word to be input into the LLM includes a keyword that matches a prompt injection word in a pre-constructed prompt injection word library, so as to detect whether the prompt word includes a prompt injection word. Keyword matching can, for example, adopt the Deterministic Finite Automaton (DFA) algorithm, which can significantly improve the retrieval efficiency in data retrieval and ensure that the detection process is fast and accurate. When it is detected through keyword matching technology that the prompt word includes a keyword that matches a prompt injection word in the prompt injection word library, the prompt injection word detection module can determine that the prompt word includes a prompt injection word; conversely, when it is detected through keyword matching technology that the prompt word does not include a keyword that matches a prompt injection word in the prompt injection word library, the prompt injection word detection module can determine that the prompt word does not include a prompt injection word.

[0083] Alternatively, in some other embodiments, prompt injection word detection can be performed through a knowledge base or a knowledge graph. For example, the prompt injection word detection module can first convert the prompt word into a structured query statement, such as Structured Query Language (SQL), so that the prompt word can be used in a more standardized form for subsequent query operations. Subsequently, the prompt injection word detection module retrieves whether the prompt word includes a prompt injection word in the knowledge base or the knowledge graph according to the query statement. For example, when it is retrieved in the knowledge base or the knowledge graph according to the query statement that the prompt word is associated with a known prompt injection pattern, the prompt injection word detection module can determine that the prompt word includes a prompt injection word; conversely, when it is retrieved in the knowledge base or the knowledge graph according to the query statement that the prompt word has no relation to a known prompt injection pattern, the prompt injection word detection module can determine that the prompt word does not include a prompt injection word.

[0084] It should be noted that step 301 and step 302 can be executed in parallel, or step 301 can be executed first and then step 302, or step 302 can be executed first and then step 301. The embodiments of the present disclosure do not limit this.

[0085] Step 303, the model fusion module analyzes whether the prompt word involves a prompt injection attack according to the semantic analysis result and the prompt injection word detection result.

[0086] The computing device running the fusion recognition model 20 uses the model fusion module 23 of the fusion recognition model 20 to analyze whether the prompt word involves a prompt injection attack based on the semantic analysis result and the prompt injection word detection result. For example, logical rules can be preset to analyze whether the prompt word involves a prompt injection attack based on the semantic analysis result and the prompt injection word detection result. For example, when the semantic analysis result is the result obtained by binarizing the probability value obtained by the neural network model using a preset binarization threshold (the semantic analysis result is that the prompt word has an abnormal semantic expression involving a prompt injection attack or the prompt word has a normal semantic expression not involving a prompt injection attack), the logical rule can be: when the semantic analysis result is that the prompt word has a normal semantic expression not involving a prompt injection attack, and the prompt injection word detection result is that the prompt word does not include the preset prompt injection word, the fusion recognition model can determine that the prompt word does not involve a prompt injection attack; when the semantic analysis result is that the prompt word has an abnormal semantic expression involving a prompt injection attack, or the prompt injection word detection result is that the prompt word includes the preset prompt injection word, the fusion recognition model can determine that the prompt word involves a prompt injection attack. This way can quickly fuse the semantic analysis result and the prompt injection word detection result to obtain a final analysis result that comprehensively considers the semantic analysis result and the prompt injection word detection result.

[0087] In some embodiments, when the probability value obtained by the neural network model is directly used as the semantic analysis result (the higher the probability value, the higher the probability that the prompt word has a normal semantic expression not involving a prompt injection attack, and the less likely the prompt word involves a prompt injection attack), in order to make the final analysis result more accurate and improve the recognition efficiency, two thresholds can be set for the probability value used as the semantic analysis result. When the probability value of the semantic analysis result is greater than or equal to the larger first threshold (for example, 0.9), it means that the probability that the prompt word has a normal semantic expression not involving a prompt injection attack is relatively high, and it can be determined that the prompt word does not involve a prompt injection attack only based on the semantic analysis result. When the probability value of the semantic analysis result is less than or equal to the smaller second threshold (for example, 0.6), it means that the probability that the prompt word has a normal semantic expression not involving a prompt injection attack is relatively low, and it can be determined that the prompt word does not involve a prompt injection attack only based on the semantic analysis result. When the probability value falls between the two thresholds, it means that the neural network model performing the semantic analysis is not very sure whether the prompt word has a normal semantic expression not involving a prompt injection attack. Therefore, it is difficult to determine whether the prompt word involves a prompt injection attack only based on the semantic analysis result, and it is necessary to further analyze whether the prompt word involves a prompt injection attack according to the prompt injection word detection result. At this time, when the prompt injection word detection result is that the prompt word does not include the prompt injection word, it can be determined that the prompt word does not involve a prompt injection attack, and when the prompt injection word detection result is that the prompt word includes the prompt injection word, it is determined that the prompt word involves a prompt injection attack.

[0088] In this case, for the method of analyzing whether a prompt involves a prompt injection attack based on the semantic analysis result and the prompt injection word detection result, for example, refer to Figure 4 . In Figure 4 , first, in step 401, it is judged whether the probability value of the semantic analysis result is greater than or equal to 0.9. When it is judged as "yes" in step 401, that is, when the probability value of the semantic analysis result is greater than or equal to 0.9, proceed to step 404, that is, determine that the prompt does not involve a prompt injection attack.

[0089] When it is judged as "no" in step 401, that is, when the probability value of the semantic analysis result is less than 0.9, proceed to step 402, that is, judge whether the probability value of the semantic analysis result is less than or equal to 0.6. When it is judged as "yes" in step 402, that is, when the probability value of the semantic analysis result is less than or equal to 0.6, proceed to step 405, that is, determine that the prompt involves a prompt injection attack.

[0090] When it is judged as "no" in step 402, that is, when the probability value of the semantic analysis result is less than 0.9 but greater than 0.6, proceed to step 403, that is, according to the prompt injection word detection result, judge whether the prompt includes a prompt injection word.

[0091] When it is judged as "no" in step 403, that is, when the prompt does not include a prompt injection word, proceed to step 404, that is, determine that the prompt does not involve a prompt injection attack. When it is judged as "yes" in step 403, that is, when the prompt includes a prompt injection word, proceed to step 405, that is, determine that the prompt involves a prompt injection attack.

[0092] It should be noted that in some embodiments, the prompt injection word detection of the prompt may not be performed first, and between step 402 and step 403, that is, when it is judged that the probability value of the semantic analysis result is less than 0.9 but greater than 0.6, the prompt injection word detection of the prompt is performed again. In this case, when the probability value of the semantic analysis result is greater than or equal to 0.9 and when the probability value of the semantic analysis result is less than or equal to 0.6, only semantic analysis needs to be performed without performing prompt injection word detection, which further improves the recognition efficiency and saves computing resources.

[0093] In some embodiments, when directly using the probability values obtained by the neural network model as the semantic analysis results (the higher the probability value, the higher the probability that the prompt word has a normal semantic expression not involving prompt injection attacks, and the less likely the prompt word involves prompt injection attacks), in order to make the final analysis results more accurate and improve the recognition efficiency, the semantic analysis results and the injection prompt word detection results can also be weighted and summed, and it is determined whether the prompt word involves prompt injection attacks based on the weighted sum. For example, the injection prompt word detection results can be represented numerically. When the injection prompt word detection result is that the prompt word includes an injection prompt word, the numerical value is set to 0, and when the injection prompt word detection result is that the prompt word does not include an injection prompt word, the numerical value is set to 1. Weights w1 and w2 are respectively assigned to the semantic analysis results and the injection prompt word detection results. The higher the weight, the higher the confidence level, that is, the more the result is trusted. When P1 represents the semantic analysis result (a probability value between 0 and 1) and P2 represents the injection prompt word detection result (0 or 1), the weighted sum can be expressed as E = w1×P1 + w2×P2. A threshold can be set for the weighted sum. When the weighted sum exceeds the threshold, it is determined that the prompt word does not involve prompt injection attacks, and when the weighted sum is lower than the threshold, it is determined that the prompt word involves prompt injection attacks. For example, when w1 and w2 are respectively set to 0.7 and 0.3, and the threshold of the weighted sum is set to 0.75, when P1 = 0.8 and P2 = 1, the weighted sum E = 0.86 > 0.75, and it can be determined that the prompt word does not involve prompt injection attacks.

[0094] In some embodiments, the semantic analysis results and the injection prompt word detection results can be fused based on a deep learning model. For example, the probability value of the semantic analysis result and the binary variable of the injection prompt word detection result can be concatenated into a feature vector, and the result of whether the manually identified prompt word involves prompt injection attacks is used as the label of the feature vector. A large number of feature vectors and their labels are input into a simple neural network to train the neural network. During application, after inputting the feature vector concatenated by the probability value of the semantic analysis result and the binary variable of the injection prompt word detection result into the trained neural network, the trained neural network can output the result of whether the prompt word involves prompt injection attacks.

[0095] Through the above possible implementation methods, the model fusion module can comprehensively consider the semantic analysis results and the detection results, more comprehensively and accurately judge whether the prompt word involves prompt injection attacks, thereby effectively improving the security and reliability of the LLM system.

[0096] In some embodiments, different treatments can be applied to the prompting words according to whether they involve prompt injection attacks. When the prompting words do not involve prompt injection attacks, the prompting words can be allowed to be input into the LLM; when the prompting words involve prompt injection attacks, the prompting words can be blocked from being input into the LLM, and a prompt message indicating that the prompting words involve prompt injection attacks can be output. The prompt message can include not only the conclusion that the prompting words input by the user involve prompt injection attacks (and have been intercepted), but also provide further guidance or suggestions. For example, in the prompt message, it can be explained why it is determined that the prompting words involve prompt injection attacks. For example, the prompt message can be "The topics before and after the prompting words you input are not unified". This can help users understand the security mechanism and judgment criteria of the system. In addition, modification suggestions can also be provided in the prompt message to assist users in adjusting the prompting words to ensure that the prompting words are safe and compliant. For example, the prompt message can be "The act of stealing a car appears in the prompting words you input, and such an act is illegal. Please delete the relevant description". In addition, users can also be informed of the correct way to input prompting words through the prompt message, such as providing relevant usage guides or examples, to promote safe and efficient interaction between users and the LLM system.

[0097] In summary, according to the characteristics of prompt injection attacks, the embodiments of the present disclosure design a comprehensive protection strategy to defend against prompt injection attacks and enhance the security of the model. Through the comprehensive analysis of the fusion recognition model, the accuracy of identifying prompt injection attacks is improved. It helps to protect the LLM from malicious attacks and ensure its stable operation. On the one hand, by combining a neural network model, a prompt injection word detection module, and a model fusion module, a multi-level defense system is constructed, improving the comprehensiveness and depth of recognition. The functions and roles of each module are clarified, making the recognition process more transparent and interpretable. By comprehensively analyzing various results, the accuracy and reliability of recognition are improved. On the other hand, the neural network model performs semantic analysis on the prompt words, and represents the possibility that the prompt words have a normal semantic expression not involving prompt injection attacks with probability values, providing a more detailed basis for judgment for recognition, helping to more accurately evaluate the security of the prompt words, and reducing false positives and false negatives. On the other hand, two methods for detecting prompt injection words are provided. One is direct matching, which detects whether the prompt word includes keywords that match the prompt injection words in the pre-constructed prompt injection word library; the other is structured query, which converts the prompt word into a structured query statement, and according to the query statement, retrieves in the knowledge base or knowledge graph whether the prompt word includes prompt injection words, increasing the flexibility and adaptability of detection. On the other hand, by setting two thresholds to comprehensively analyze the semantic analysis results and the prompt injection word detection results, a refined recognition strategy is provided, improving the accuracy of recognition. On the other hand, different processing measures are taken for the prompt words according to whether the prompt words involve prompt injection attacks. For example, when the prompt word does not involve prompt injection attacks, the prompt word is allowed to be input into the LLM; when the prompt word involves prompt injection attacks, the prompt word is blocked from being input into the LLM, and a prompt message indicating that the prompt word involves prompt injection attacks is output, improving the response ability and security of the system. By outputting the prompt message, the user experience is enhanced, enabling users to understand why their prompt words are rejected and guiding them on how to improve. In short, this comprehensive protection strategy not only effectively defends against prompt injection attacks, but also brings significant improvements to the security, robustness, and user satisfaction of the system.

[0098] It should be noted that when the device provided in the above embodiment realizes its functions, only the above-mentioned division of each functional module is used for illustration. In practical applications, the above functions can be allocated to different functional modules according to actual needs, that is, the content structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0099] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment related to the method, and will not be elaborated here.

[0100] An embodiment of the present disclosure also provides a computing device, including a memory, a processor, and a computer program stored on the memory, where the processor executes the computer program to implement the operations of the above method.

[0101] An embodiment of the present disclosure also provides a non-volatile computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the operations of the above method are implemented.

[0102] An embodiment of the present disclosure also provides a computer program product, including a computer program, or a non-volatile computer-readable storage medium carrying the computer program, and when the computer program is executed by a processor, the operations of the above method are implemented.

[0103] Figure 5 is a block diagram of a device 1900 shown according to an exemplary embodiment. For example, the device 1900 is used to execute the method in the above embodiment and may be provided as a server or a terminal device. Referring to Figure 5 , the device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by a memory 1932 for storing instructions executable by the processing component 1922, such as application programs. The application programs stored in the memory 1932 may include one or more modules each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0104] The device 1900 may further include a power supply component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and an input / output interface 1958 (I / O interface). The device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server TM , MacOS X TM , Unix TM , Linux TM , FreeBSD TM or the like.

[0105] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, and the above computer program instructions can be executed by the processing component 1922 of the device 1900 to complete the above method.

[0106] A computer-readable storage medium can be a tangible device that can retain and store programs / instructions used by an instruction execution device. The computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device, such as a punched card or raised structures in grooves storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage medium used herein is not construed as being an instantaneous signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated through a waveguide or other transmission medium (e.g., an optical pulse through an optical fiber cable), or an electrical signal transmitted through a wire.

[0107] The computer programs (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to respective computing / processing devices, or downloaded to an external computer or an external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include a copper transmission cable, an optical fiber transmission, a wireless transmission, a router, a firewall, a switch, a gateway computer, and / or an edge server. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing / processing device.

[0108] A computer program (or computer program instructions) for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine - related instructions, microcode, firmware instructions, state - setting data, or source code or object code written in any combination of one or more programming languages, including object - oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer - readable program instructions may be executed entirely on the user's computer, partially on the user's computer, executed as a stand - alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via an Internet service provider through the Internet). In some embodiments, by using the state information of the computer - readable program instructions to customize an electronic circuit, such as a programmable logic circuit, a field - programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer - readable program instructions to implement various aspects of the present disclosure.

[0109] Aspects of the present disclosure are described herein with reference to the flowchart and / or block diagram of a method, apparatus (system), and computer program product according to embodiments of the present disclosure. It should be understood that each block of the flowchart and / or block diagram, and the combinations of blocks in the flowchart and / or block diagram, can be implemented by computer - readable program instructions.

[0110] These computer - readable program instructions can be provided to a processor of a general - purpose computer, a special - purpose computer, or other programmable data - processing apparatus to produce a machine such that, when the instructions are executed by the processor of the computer or other programmable data - processing apparatus, a device is produced that implements the functions / acts specified in one or more blocks of the flowchart and / or block diagram. These computer - readable program instructions can also be stored in a computer - readable storage medium, and these instructions cause a computer, a programmable data - processing apparatus, and / or other devices to work in a particular manner. Thus, the computer - readable medium storing the instructions includes a manufacture that includes instructions for implementing various aspects of the functions / acts specified in one or more blocks of the flowchart and / or block diagram.

[0111] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices, causing a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other devices to produce a computer-implemented process such that the instructions executed on the computer, other programmable data processing apparatus, or other devices implement the functions / acts specified in one or more boxes of the flowchart and / or block diagram.

[0112] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two consecutive blocks may in fact be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or acts, or by a combination of dedicated hardware and computer instructions.

[0113] The embodiments of the present disclosure have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, the practical application, or improvements made to the technology in the market, or to enable other ordinary skilled persons in the art to understand the embodiments disclosed herein.

Claims

1. A method for identifying prompt injection attacks against large language models (LLMs), characterized in that, The method includes: Obtaining a prompt to be input into the LLM; Performing semantic analysis on the prompt to obtain a semantic analysis result; Performing prompt injection word detection on the prompt to obtain a prompt injection word detection result; Analyzing whether the prompt involves a prompt injection attack based on the semantic analysis result and the prompt injection word detection result.

2. The method according to claim 1, wherein The performing semantic analysis on the prompt includes: Using a trained neural network model to perform context-based semantic analysis on the prompt.

3. The method according to claim 2, wherein The performing prompt injection word detection on the prompt includes: Detecting whether the prompt includes a keyword that matches a prompt injection word in a pre-constructed prompt injection word library, or Converting the prompt into a structured query statement and retrieving whether the prompt includes a prompt injection word in a knowledge base or a knowledge graph according to the query statement.

4. The method according to claim 3, characterized in that The semantic analysis result includes a probability value that the prompt has a normal semantic expression not involving a prompt injection attack, and Analyzing whether the prompt involves a prompt injection attack based on the semantic analysis result and the prompt injection word detection result includes: When the probability value is greater than or equal to a first threshold, determining that the prompt does not involve a prompt injection attack, When the probability value is less than or equal to the second threshold, determining that the prompt involves a prompt injection attack, where the second threshold is less than the first threshold, and When the probability value is less than the first threshold and greater than the second threshold, further analyzing whether the prompt involves a prompt injection attack according to the prompt injection word detection result.

5. The method according to claim 4, wherein Further analyzing whether the prompt involves a prompt injection attack according to the prompt injection word detection result includes: When the prompt does not include a prompt injection word, determining that the prompt does not involve a prompt injection attack, and When the prompt includes a prompt injection word, determining that the prompt involves a prompt injection attack.

6. The method according to any one of claims 1 to 5, characterized in that The method further includes: When the prompt does not involve a prompt injection attack, allowing the prompt to be input into the LLM; When the prompt involves a prompt injection attack, blocking the prompt from being input into the LLM and outputting a prompt message indicating that the prompt involves a prompt injection attack.

7. A fusion recognition model for identifying prompt injection attacks against LLMs, characterized in that, The fusion recognition model includes: A neural network model for performing semantic analysis on a prompt to be input into the LLM to obtain a semantic analysis result; A prompt injection word detection module for performing prompt injection word detection on the prompt to obtain a prompt injection word detection result; A model fusion module for analyzing whether the prompt involves a prompt injection attack based on the semantic analysis result and the prompt injection word detection result.

8. A computing device, comprising a memory, a processor, and a computer program stored on the memory, characterized in that, The processor executes the computer program to implement the operations of the method according to any one of claims 1 to 6.

9. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the operations of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program or a non-volatile computer-readable storage medium carrying the computer program, characterized in that, When the computer program is executed by the processor, it implements the operations of the method according to any one of claims 1 to 6.

Images