Context learning-based large language model prompt word injection attack detection method and device

Abstract

The application discloses a large language model prompt word injection attack detection method and device based on context learning, and belongs to the technical field of artificial intelligence, and comprises the following steps: based on a Bert pre-training model, inherent features and dependency relationships between different levels and different labels are learned by introducing a label-based attention module, a multi-level, multi-label and fine-grained classification model for prompt word injection attack is designed, and accurate identification of the prompt word injection attack is realized.Meanwhile, according to the context learning, the prediction ability of the classification model is combined with the ability of the large language model, and the defense ability of the large language model to the prompt word injection attack is improved.The application can automatically detect the prompt word injection attack, improve the effectiveness and comprehensiveness of detection, and can be effectively applied to the field of large model security detection.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a method and device for detecting prompt word injection attacks on a large language model based on context learning. Background Art

[0002] Large language models are one of the important research directions in the field of artificial intelligence in recent years. With their excellent natural language processing capabilities, large language models have gained widespread attention and popularity, and have completely changed the field of natural language processing.

[0003] Although large language models have achieved great success in many application areas, they have also exposed some serious security issues during their application. Currently, researchers have designed a considerable number of instructions for attacking large language models.

[0004] The most common attack method is prompt word injection, which includes jailbreak attacks, target hijacking attacks, and prompt word leakage attacks. Researchers have found that by using carefully crafted attack commands as input, large language models can be caused to generate inappropriate content, leak sensitive information, and even behave abnormally.

[0005] To address the above issues, the need for detecting large language model prompt word injection attack instructions is becoming increasingly prominent. Existing detection methods for large model prompt word injection attack instructions mainly rely on manual detection, but this method has the following limitations:

[0006] 1. Poor scalability: As large language models continue to evolve, previous test cases may not be directly applicable to new versions of the model. This makes it challenging for manual detection methods to maintain their effectiveness.

[0007] 2. High cost: Manual safety testing requires a lot of money and time investment, which increases the cost of the testing process.

[0008] In addition, due to the wide variety of cue word injection attacks, the existing defense strategies of large language models, including keyword matching and semantic detection strategies, cannot effectively defend against cue word injection attacks. Current defense methods are also unable to achieve unified and fine-grained detection and classification of cue word injection attacks.

[0009] To address these issues, patent publication CN112860889A discloses a BERT-based multi-label classification method, comprising a data preprocessing module, a BERT fine-tuning module, and a classifier module. The sentences and all labels in the text are grouped into sentence pairs. Leveraging the BERT model's significant advantages in sentence pair classification across multiple domains, the sentence vectors representing the sentences and labels are enriched with contextual semantic information. Finally, the resulting sentence vectors are passed to the classifier module to determine the semantic relationship between the sentence and the label, thereby predicting whether the sentence is labeled with that label.

[0010] This invention uses the BERT model for sentence classification tasks, and uses the BERT model to reveal the contextual relationship between sentences to understand the information contained in the sentences. However, this invention does not mention the judgment of the prompt word injection attack category for large language models.

[0011] The patent document with publication number CN117278273A (publication date 2023-12-22) discloses a detection method for SQL injection attacks based on BERT and LSTM, including: step one, data collection and data normalization; step two, tokenizer encoding preprocessing of positive samples and negative samples; step three, using the BERT and LSTM combined network to train the detection model, and obtaining the final detection model after the training is completed; step four, detecting SQL injection attacks through the detection model. If the result is a malicious attack, an alarm is issued, and all detection results are recorded and logged.

[0012] This invention combines BERT and LSTM to capture contextual information and sequence dependencies in text and achieve SQL injection attack detection. However, since SQL injection attacks mainly focus on inserting malicious SQL code into database queries, while prompt word injection attacks may involve malicious tampering or misleading of model input data, that is, SQL injection attacks and prompt word injection attacks differ in type and mechanism. Therefore, applying the SQL injection detection method proposed in this invention to prompt word injection detection cannot fully cover all attack scenarios.

[0013] To address these issues, it is necessary to develop smarter automated tools and methods to improve the detection accuracy of cue word injection attack instructions for large language models. Summary of the Invention

[0014] The purpose of the present invention is to provide a large language model prompt word injection attack detection method based on context learning. Through a context detection method combining a BERT-based hierarchical multi-label classification model and a large language model, the prompt word injection attack type of the input text sequence is predicted, and the cause analysis of the prediction process is given. The method of the present invention can realize automatic detection of text and has a high degree of interpretability.

[0015] To achieve the above-mentioned purpose, the present invention provides the following technical solutions:

[0016] In a first aspect, an embodiment of the present invention provides a method for detecting a large language model prompt word injection attack based on context learning, comprising the following steps:

[0017] Step 1: Based on the language mutation algorithm of fuzz testing, the prompt word injection attack instruction is mutated to obtain the mutated attack instruction. The large language model is cyclically attacked through the mutated attack instruction, and the mutated attack instructions that are effective are retained to construct the instruction dataset.

[0018] Step 2: Design a multi-level, multi-label fine-grained classification tree. Based on the attack purpose of the prompt word injection attack command, construct the first level of the fine-grained classification tree. Perform semantic segmentation on the first level to obtain the second level. According to the implementation strategy of each label in the second level, obtain the third level. Use the fine-grained classification tree to add prompt word injection attack category labels to the instruction dataset.

[0019] Step 3: Based on the BERT pre-trained model and fine-grained classification tree, a label-based attention module is introduced to build a hierarchical multi-label classification model. The fine-grained classification tree is used to stratify the input text and assign hierarchical prediction labels; the BERT pre-trained model is used to obtain the hierarchical embedding vector of the hierarchical text; the label-based attention module is used to extract the label features and hierarchical features of the hierarchical embedding vector, and the weighted average is used as the confidence of the hierarchical prediction label of the corresponding layer. The hierarchical prediction label corresponding to the highest confidence above the threshold is integrated as the predicted label of the input text;

[0020] Step 4: Fine-tune the hierarchical multi-label classification model with the labeled instruction dataset to obtain a fine-tuned prediction model. Obtain a sampling sample of the labeled instruction dataset. Use the fine-tuned prediction model to construct a context set and test prompts for the prediction output of the sampling sample and test sample respectively. After splicing, input them into the large language model and use context learning to obtain the prediction results and cause analysis of the test sample.

[0021] The technical concept of this invention is to propose a method for detecting cue word injection attacks in large language models based on contextual learning and hierarchical multi-label classification. Based on a BERT pre-trained model and a fine-grained classification tree, this method introduces a label-based attention module to construct a hierarchical multi-label classification model for predicting labels and confidence levels for input text.

[0022] Among them, by constructing a multi-level fine-grained classification tree containing specific prompt word injection attack categories, the input text is stratified according to the fine-grained classification tree and potential hierarchical prediction labels are assigned to different levels.

[0023] The BERT model then converts each level of text into a hierarchical embedding vector. The label-based attention module learns the label and hierarchical features of each level. A multi-layer perceptron is used to convert the extracted label and hierarchical features into local and global prediction vectors, respectively. A weighted average of these two vectors is then taken to generate a confidence vector. The elements in the confidence vector represent the confidence of the hierarchical prediction label for each level of text in the fine-grained classification tree. By transferring hierarchical features from different levels, the dependencies between features at different levels are learned.

[0024] At the same time, the present invention proposes for the first time a method for detecting prompt word injection attack categories by combining a BERT-based hierarchical multi-label classification model and a large language model for contextual learning, combining the predictive ability of the hierarchical multi-label classification model with the capabilities of the large language model.

[0025] The predicted label and confidence of the sampled samples are obtained through a hierarchical multi-label classification model. A prompt template in the form of sample-predicted label-confidence-true label is constructed, and a test prompt in the form of sample-predicted label-confidence is constructed. The prompt template and test prompt are spliced ​​and input into the large language model. The large language model learns the form of the prompt template, that is, through context learning, the true label of the test sample is output in the same form, and the cause of the prediction process of the true label is analyzed.

[0026] Furthermore, in step 1, the language mutation algorithm based on fuzz testing mutates the prompt word injection attack instruction, and the mutation method includes style transformation, sentence expansion or scene nesting.

[0027] Furthermore, in step 2, the fine-grained classification tree, each level specifically includes:

[0028] According to the attack purpose, the first level includes: jailbreak attack, target hijacking attack, and prompt word leakage attack;

[0029] The first level is semantically divided, and the second level includes: impersonation, privilege escalation, attention diversion, system prompt word leakage, and user prompt word leakage;

[0030] Based on the implementation strategies of the second level, the third level includes: role-playing, responsibility allocation, research and experimentation, special modes, advanced modes, language translation, text continuation, and specified formats. By classifying existing cue word injection attacks and constructing a fine-grained classification tree, we use this tree to assign clear, fine-grained hierarchical prediction labels to the input text, thus avoiding the problem of inability to accurately classify existing cue word injection attack detection methods.

[0031] Furthermore, in step 3, the weighted average of the obtained label features and hierarchical features is used as the confidence, and the hierarchical prediction labels corresponding to the highest confidence above the threshold are integrated as the predicted labels of the input text, specifically:

[0032] The label features and layer features of each layer are respectively passed through a multi-layer perceptron to generate a label-based local prediction vector and a layer-based global prediction vector, where the elements in the local prediction vector and the global prediction vector are the local prediction probability and the global prediction probability respectively;

[0033] The confidence vector of the hierarchical prediction label is obtained by weighted averaging the elements at corresponding positions in the local prediction vector and the global prediction vector, where the elements in the confidence vector are the confidences of the hierarchical prediction labels of the input text by the fine-grained classification tree;

[0034] The level prediction label corresponding to the highest confidence level above the threshold in the confidence vector is selected as the prediction label of the corresponding level;

[0035] The predicted labels of each level are integrated as the predicted labels of the input text.

[0036] Furthermore, in step 3, after the label-based attention module extracts the label features and hierarchical features of the hierarchical embedding vector, the hierarchical features of the previous layer are input into the label-based attention module of the next layer to extract the label features and hierarchical features together with the hierarchical embedding vector of the next layer. By passing the hierarchical features of the previous layer to the next layer, the dependencies between different layers are learned, thereby improving the accuracy of the predicted labels.

[0037] Furthermore, in step 4, the context set and test prompt are constructed respectively based on the prediction output of the sampling sample and the test sample using the fine-tuning prediction model, specifically:

[0038] Input the sample into the fine-tuned prediction model to obtain a first predicted label and a first confidence level. Based on the predicted label, confidence level, and label of the sample, an example template in the form of sample-predicted label-confidence level-true label is constructed and stored in the context set.

[0039] The test sample is fed into the fine-tuned prediction model to obtain a second predicted label and a second confidence level. Based on the example template, a test prompt is constructed in the form of sample-predicted label-confidence level. The context set and test prompt are then fed into the subsequent large language model. By constructing a contextual learning network that combines the fine-tuned prediction model with the large language model, the large language model becomes more robust against category-specific prompt word injection attacks.

[0040] Furthermore, in step 4, the concatenated data is input into the large language model, and context learning is used to obtain the prediction results and cause analysis of the test samples, specifically:

[0041] After concatenating the context set and the test prompt, they are input into the large language model together with the cause analysis prompt words;

[0042] The large language model learns the example template corresponding to the context set, obtains the true label of the test sample based on the predicted label and confidence level corresponding to the test sample in the test prompt, and outputs the prediction result of the test sample in the form of the example template. The prediction result is the sample-predicted label-confidence-true label of the test sample;

[0043] The large language model uses the cause analysis prompt word to analyze the reasoning process of the test sample's true label. Leveraging the large language model's natural language capabilities, it provides a semantic explanation of the detection process of the input text's true label, improving the interpretability of the large language model's method for detecting prompt word injection attacks.

[0044] In a second aspect, to achieve the above-mentioned purpose of the invention, an embodiment of the present invention further provides a large language model prompt word injection attack detection device based on context learning, comprising an instruction data set construction unit, a fine-grained classification tree construction unit, a classification model construction unit, and a context prediction unit;

[0045] The instruction data set construction unit is used to mutate the prompt word injection attack instruction based on the language mutation algorithm of fuzz testing to obtain the mutated attack instruction, perform a cyclic attack on the large language model through the mutated attack instruction, retain the mutated attack instruction that is effective, and construct the instruction data set;

[0046] The fine-grained classification tree construction unit is used to design a multi-level and multi-label fine-grained classification tree. According to the attack purpose of the prompt word injection attack instruction, the first level of the fine-grained classification tree is constructed, the first level is semantically divided to obtain the second level, and the third level is obtained according to the implementation strategy of each label of the second level; the fine-grained classification tree is used to add the prompt word injection attack category label to the instruction data set;

[0047] The classification model construction unit is used to introduce a label-based attention module based on the BERT pre-training model and the fine-grained classification tree to construct a hierarchical multi-label classification model, wherein the fine-grained classification tree is used to stratify the input text and assign hierarchical prediction labels; the BERT pre-training model is used to obtain the hierarchical embedding vector of the hierarchical text; the label-based attention module is used to extract the label features and hierarchical features of the hierarchical embedding vector, and use the weighted average as the confidence of the hierarchical prediction label of the corresponding layer, and integrate the hierarchical prediction label corresponding to the highest confidence above the threshold as the prediction label of the input text;

[0048] The context prediction unit is used to fine-tune the hierarchical multi-label classification model with a labeled instruction dataset to obtain a fine-tuned prediction model, obtain sampling samples of the labeled instruction dataset, and use the fine-tuned prediction model to construct a context set and test prompts for the prediction outputs of the sampling samples and test samples respectively. After splicing, they are input into the large language model, and context learning is used to obtain the prediction results and cause analysis of the test samples.

[0049] In the third aspect, in order to achieve the above-mentioned purpose of the invention, an embodiment of the present invention also provides a large language model prompt word injection attack detection device based on context learning, including a memory and a processor, the memory is used to store a computer program, and the processor is used to implement the large language model prompt word injection attack detection method based on context learning provided by the embodiment of the present invention in the first aspect when executing the computer program.

[0050] In a fourth aspect, in order to achieve the above-mentioned purpose of the invention, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program uses a computer, the large language model prompt word injection attack detection method based on context learning provided by the embodiment of the present invention in the first aspect is implemented.

[0051] The beneficial effects of the present invention are as follows:

[0052] (1) The present invention classifies prompt word injection attack instructions and constructs a multi-level, multi-label fine-grained classification tree. The fine-grained classification tree is used to label the input text sequence at multiple levels, and each level is given a specific and clear level prediction label, thereby avoiding the problem of existing prompt word injection attack sample detection methods that only detect risks but do not provide specific risk categories.

[0053] (2) Based on the BERT pre-training model and fine-grained classification tree, the present invention introduces a label-based attention module and designs a multi-level, multi-label fine-grained classification model for prompt word injection attacks to achieve accurate identification of prompt word injection attacks. The hierarchical multi-label classification model is used to extract the label features and hierarchical features of the text, and the hierarchical features of the previous level are transferred to the next level to learn the dependency relationship between different levels, thereby achieving more accurate identification and classification of prompt word injection attack instructions;

[0054] (3) In the context learning process, the present invention combines a BERT-based hierarchical multi-label classification model with a large language model to detect prompt word injection attacks. The hierarchical multi-label classification model is used to obtain the predicted labels and confidences of the test samples, and a context set of the sampled samples is constructed as the prior knowledge of the large language model. The large language model infers the true labels of the test samples based on the prior knowledge, the predicted labels and confidences of the test samples, and transfers the prediction performance of the hierarchical multi-label classification model to the large language model, thereby improving the prediction accuracy and interpretability of the large language model, making the large language model more robust in the face of prompt word injection attacks.

[0055] (4) The present invention combines the fine-tuned hierarchical multi-label classification model with the large language model, which can automatically detect prompt word injection attacks while improving the effectiveness and comprehensiveness of the detection, and can be effectively applied to the field of large model security detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a flowchart of a method for detecting a large language model prompt word injection attack based on context learning provided by an embodiment of the present invention.

[0057] Figure 2 It is a multi-level, multi-label, fine-grained classification tree provided by an embodiment of the present invention.

[0058] Figure 3 It is a structural diagram of a classification network provided by an embodiment of the present invention.

[0059] Figure 4 This is a diagram of the detection and analysis process using a large language model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0061] like Figure 1As shown, the embodiment provides a method for detecting a large language model prompt word injection attack based on context learning, including the following steps:

[0062] S110, based on the language mutation algorithm of fuzz testing, mutates the prompt word injection attack instruction to obtain the mutated attack instruction, performs a cyclic attack on the large language model through the mutated attack instruction, retains the mutated attack instruction that is effective, and constructs an instruction data set.

[0063] This embodiment uses a fuzz-based language mutation algorithm to mutate instructions. Fuzz-based language mutation algorithms are primarily used in software testing, particularly to discover potential vulnerabilities and errors in programming language implementations or parsers. These algorithms test the target program by generating random or mutated input data and observing its responses to detect abnormal behavior. Successful mutated instructions are retained to construct a dataset of prompt word injection attack instructions. The specific process is as follows:

[0064] 1.1) Collect the original prompt words and inject them into the attack instruction seed pool through network collection, manual design and other methods.

[0065] 1.2) Using a language mutation algorithm based on fuzz testing, the prompt word injection attack instructions in the original prompt word injection attack instruction seed pool are mutated through mutation operations such as style transformation, sentence expansion, and scene nesting. The mutated attack instructions are stored in the prompt word injection attack instruction dataset.

[0066] 1.3) Use the attack commands from the dataset to attack the large language model. If the attack is successful, the attack command is retained; otherwise, it is deleted. After repeated iterations, the final command dataset is obtained.

[0067] S120, design a multi-level and multi-label fine-grained classification tree, construct the first level of the fine-grained classification tree according to the attack purpose of the prompt word injection attack instruction, perform semantic division on the first level to obtain the second level, and obtain the third level according to the implementation strategy of each label of the second level; use the fine-grained classification tree to add the prompt word injection attack category label to the instruction data set.

[0068] Considering that most current risk control platforms only detect whether input contains risks but do not identify specific risk categories, this invention constructs a fine-grained classification tree that includes clear hint word injection attack categories. This allows the final prediction results to include accurate label categories, thereby enhancing interpretability and reliability. The specific process is as follows:

[0069] 2.1) If Figure 2 As shown in Figure 1, at the first level, common prompt word injection attacks can be divided into three categories: jailbreak attacks, target hijacking attacks, and prompt word leakage attacks.

[0070] 2.2) At the second level, jailbreak attacks can be roughly divided into three categories based on the semantics of the attack commands: impersonation, privilege escalation, and attention diversion. Prompt word leakage attacks can be divided into two categories: system prompt word leakage and user prompt word leakage.

[0071] 2.3) At the third level, the jailbreak attacks of the pretend category include three implementation strategies: role-playing, responsibility allocation, and research experiments. The jailbreak attacks of the privilege escalation category include two implementation strategies: special mode and advanced model. The jailbreak attacks of the attention transfer category include three implementation strategies: language translation, text continuation, and specified format.

[0072] In the constructed fine-grained classification tree, the labels at each level correspond to the possible prompt word injection attack categories of the input text. That is, the labels in the fine-grained classification tree are used as the candidate library, the input text is stratified according to the fine-grained classification tree, and the possible hierarchical prediction labels of each layer are given according to the labels in the candidate library.

[0073] S130, based on the Bert pre-training model and the fine-grained classification tree, introduces a label-based attention module to construct a hierarchical multi-label classification model, wherein the fine-grained classification tree is used to stratify the input text and assign hierarchical prediction labels; the Bert pre-training model is used to obtain the hierarchical embedding vector of the hierarchical text; the label-based attention module is used to extract the label features and hierarchical features of the hierarchical embedding vector, and the weighted average is used as the confidence of the hierarchical prediction label of the corresponding layer, and the hierarchical prediction label corresponding to the highest confidence above the threshold is integrated as the prediction label of the input text.

[0074] like Figure 3 As shown, this embodiment introduces a label-based attention module based on the Bert pre-training model and the fine-grained classification tree to construct a hierarchical multi-label classification model.

[0075] 3.1) Input text is fed into the hierarchical multi-label classification model. The input text is first divided into hierarchies using a fine-grained classification tree, and each layer is assigned a possible hierarchical prediction label. Based on the multiple prompt word injection attack labels contained in the fine-grained classification tree constructed in this embodiment, the hierarchical prediction label of the input text may be any one or more of the N types in the fine-grained classification tree.

[0076] 3.2) Use the Bert pre-trained encoder to convert each level of text into a vector representation and obtain the level embedding vector E of each level.

[0077] 3.3) A label-based attention module is set up at each level to extract features from the level embedding vectors of each level and obtain the label feature G and level feature D' of each level.

[0078] It should be noted that the label-based attention module has two inputs: the hierarchical embedding vector E and the hierarchical feature D' output from the previous layer. It also has two outputs: the hierarchical feature D' and the label feature G. By fusing the hierarchical feature D' of the previous layer—that is, inputting the hierarchical features of the previous layer into the next layer along with the hierarchical embedding vector of the next layer into the attention-based module for feature extraction—the label-based attention module can further learn the connections and constraints between layers in the multi-level classification.

[0079] 3.4) The hierarchical features D' and label features G output by the label-based attention module are further processed by the multi-layer perceptron MLP to generate a label-based local prediction vector and a hierarchical global prediction vector, respectively. The local prediction vector and the global prediction vector are both one-dimensional vectors of 1*N dimensions, and the elements in the two are the local prediction probability p l and the global prediction probability p g .

[0080] 3.5) By taking a weighted average of the elements at corresponding positions in the local prediction vector and the global prediction vector, a confidence vector is obtained, which describes the confidence of the corresponding potential layer prediction label. Similarly, the confidence vector is also a one-dimensional vector of 1*N dimensions, where the N elements correspond to the confidence p of the N types of layer prediction labels. The weighting method is as follows. In this embodiment, the hyperparameter α is set to 0.5:

[0081] p=(1-α)·p l +α·p g

[0082] The hierarchical prediction label corresponding to the highest confidence level above the threshold is used as the prediction label of this level. By setting an appropriate preset threshold, the prediction accuracy of the model can be improved during the training phase, and those prediction results with lower confidence levels can be filtered out to improve the accuracy of the model.

[0083] like Figure 3 As shown, taking the hierarchical fine-grained classification model containing three levels constructed in this embodiment as an example, the prediction labels of each level are integrated to obtain the prediction labels of the input text. The prediction labels of the input text are expressed as: first level: prediction label 1; second level: prediction label 2; third level: prediction label 3.

[0084] S140, fine-tune the hierarchical multi-label classification model with the labeled instruction dataset to obtain a fine-tuned prediction model, obtain sampling samples of the labeled instruction dataset, use the fine-tuned prediction model to construct context sets and test prompts for the prediction outputs of the sampling samples and test samples respectively, splice them and input them into the large language model, use context learning to obtain the prediction results and cause analysis of the test samples.

[0085] 4.1) If Figure 4 As shown, first, the hierarchical multi-label classification model constructed above is fine-tuned on the constructed prompt word injection attack instruction dataset D to obtain the fine-tuned model P.

[0086] 4.2) Perform random sampling: Randomly select several samples from dataset D and store them in set R. For each sample in R, use the fine-tuned model P to make a prediction, obtaining a predicted label and confidence score. Construct a prompt template of sample-predicted label-confidence score-true label, and store the prediction results of the sampled samples in the context set C in the form of a uniform prompt template.

[0087] 4.3) For each input test sample, obtain its predicted label and confidence score in the same way, and construct a test prompt T of sample-prediction label-confidence. Concatenate C and T to obtain the final complete input I.

[0088] 4.4) I and the cause analysis prompt are input into the large language model M. The large language model first learns the prompt template in the context set and, based on the sample, predicted label, and confidence level in the test prompt, obtains the true label corresponding to the test sample. The prediction result for the test sample is output in the form of the prompt template. The output format of the prediction result is the sample - predicted label - confidence level - true label. Simultaneously, the large language model provides a corresponding explanation of the reasoning process of the true label based on the cause analysis prompt.

[0089] The prompt word injection attack detection method proposed in the present invention can be used as a large language model risk control detection module. Specifically, when a user faces a malicious request, the method proposed in the present invention will first be used to predict the label of the input sample. If the predicted true label is a jailbreak attack, the large language model will eventually output something like "Sorry, a jailbreak attack was detected in the input sample. The specific reason is (cause analysis)" based on the predicted result and cause analysis, to warn the user what type of prompt word injection attack is contained in the input sample and take corresponding defensive measures.

[0090] Based on the same inventive concept, an embodiment of the present invention also provides a large language model prompt word injection attack detection device based on context learning, including an instruction data set construction unit, a fine-grained classification tree construction unit, a classification model construction unit, and a context prediction unit;

[0091] Among them, the instruction data set construction unit is used to mutate the prompt word injection attack instruction based on the language mutation algorithm of fuzz testing to obtain the mutated attack instruction, and then cyclically attack the large language model through the mutated attack instruction, retaining the mutated attack instruction that is effective, and constructing the instruction data set;

[0092] The fine-grained classification tree construction unit is used to design a multi-level, multi-label fine-grained classification tree. Based on the attack purpose of the prompt word injection attack instruction, the first level of the fine-grained classification tree is constructed. The first level is semantically divided to obtain the second level. According to the implementation strategy of each label in the second level, the third level is obtained. The fine-grained classification tree is used to add the prompt word injection attack category label to the instruction dataset.

[0093] The classification model construction unit is used to introduce a label-based attention module based on the BERT pre-trained model and the fine-grained classification tree to build a hierarchical multi-label classification model. The fine-grained classification tree is used to stratify the input text and assign hierarchical prediction labels. The BERT pre-trained model is used to obtain the hierarchical embedding vector of the hierarchical text. The label-based attention module is used to extract the label features and hierarchical features of the hierarchical embedding vector, and the weighted average is used as the confidence of the hierarchical prediction label of the corresponding layer. The hierarchical prediction label corresponding to the highest confidence above the threshold is integrated as the prediction label of the input text.

[0094] The context prediction unit is used to fine-tune the hierarchical multi-label classification model with a labeled instruction dataset to obtain a fine-tuned prediction model, obtain sampling samples of the labeled instruction dataset, and use the fine-tuned prediction model to construct a context set and test prompts for the prediction outputs of the sampling samples and test samples respectively. After splicing, they are input into the large language model, and context learning is used to obtain the prediction results and cause analysis of the test samples.

[0095] As for the context-learning-based large language model prompt word injection attack detection device provided by the embodiment of the present invention, since it basically corresponds to the method embodiment, the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying any creative work.

[0096] Based on the same inventive concept, an embodiment also provides a large language model prompt word injection attack detection device based on context learning, including a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to implement the above-mentioned large language model prompt word injection attack detection method based on context learning when executing the computer program.

[0097] The context-learning-based large language model prompt word injection attack detection device proposed in the embodiment of the present invention can be a device such as a computer. The device embodiment can be implemented through software, and can also be implemented through hardware or a combination of software and hardware. Taking software implementation as an example, it is formed by the processor of any device with data processing capabilities in which it is located, reading the corresponding computer program instructions in the non-volatile memory into the memory and running them. From the hardware level, in addition to the processor, memory, network interface, and non-volatile memory, the context-learning-based large language model prompt word injection attack detection device provided in the embodiment of the present invention can generally include other hardware according to the actual functions of the device with data processing capabilities, which will not be described in detail.

[0098] Based on the same inventive concept, an embodiment further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is used on a computer, the computer program implements the above-mentioned context-learning-based large language model prompt word injection attack detection method.

[0099] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. Furthermore, the computer-readable storage medium may include both an internal storage unit of any device with data processing capabilities and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0100] It should be noted that the large language model prompt word injection attack detection device based on context learning, the large language model prompt word injection attack detection equipment based on context learning and the computer-readable storage medium provided in the above embodiments all belong to the same concept as the embodiment of the large language model prompt word injection attack detection method based on context learning. The specific implementation process is detailed in the embodiment of the large language model prompt word injection attack detection method based on context learning, which will not be repeated here.

[0101] The above description is only a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the implementation process of the present invention is described in detail above, it is still possible for those familiar with the art to modify the technical solutions described in the above examples or to replace some of the technical features therein with equivalents. Any modifications, equivalent replacements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A method for detecting prompt word injection attacks using a large language model based on contextual learning, characterized in that: The following steps are involved: Step 1: Based on the language mutation algorithm of fuzz testing, the prompt word injection attack instruction is mutated to obtain the mutated attack instruction. The large language model is cyclically attacked through the mutated attack instruction, and the mutated attack instructions that are effective are retained to construct the instruction dataset. Step 2: Design a multi-level, multi-label, fine-grained classification tree. Based on the attack purpose of the prompt word injection attack command, construct the first level of the fine-grained classification tree. Perform semantic segmentation on the first level to obtain the second level. According to the implementation strategy of each label in the second level, obtain the third level. Use fine-grained classification trees to add prompt word injection attack category labels to the instruction dataset; Step 3: Based on the BERT pre-trained model and fine-grained classification tree, a label-based attention module is introduced to build a hierarchical multi-label classification model. The fine-grained classification tree is used to stratify the input text and assign hierarchical prediction labels. The BERT pre-training model is used to obtain the hierarchical embedding vector of the hierarchical text; the label-based attention module is used to extract the label features and hierarchical features of the hierarchical embedding vector, and the weighted average is used as the confidence of the hierarchical prediction label of the corresponding layer. The hierarchical prediction label corresponding to the highest confidence above the threshold is integrated as the predicted label of the input text; Step 4: Fine-tune the hierarchical multi-label classification model with the labeled instruction dataset to obtain a fine-tuned prediction model. Obtain a sampling sample of the labeled instruction dataset. Use the fine-tuned prediction model to construct a context set and test prompts for the prediction output of the sampling sample and test sample respectively. After splicing, input them into the large language model and use context learning to obtain the prediction results and cause analysis of the test sample.

2. The method for detecting a large language model prompt word injection attack based on context learning according to claim 1 is characterized in that: In step 1, the language mutation algorithm based on fuzz testing mutates the prompt word injection attack instruction, and the mutation methods include style transformation, sentence expansion or scene embedding.

3. The method for detecting a large language model prompt word injection attack based on context learning according to claim 1 is characterized in that: In step 2, the fine-grained classification tree, each level specifically includes: According to the attack purpose, the first level includes: jailbreak attack, target hijacking attack, and prompt word leakage attack; The first level is semantically divided, and the second level includes: impersonation, privilege escalation, attention diversion, system prompt word leakage, and user prompt word leakage; Based on the implementation strategy of the second level, the third level includes: role playing, responsibility allocation, research and experimentation, special mode, advanced mode, language translation, text continuation and specified format.

4. The method for detecting a large language model prompt word injection attack based on contextual learning according to claim 1 is characterized in that: In step 3, the weighted average of the obtained label features and hierarchical features is used as the confidence, and the hierarchical prediction labels corresponding to the highest confidence above the threshold are integrated as the predicted labels of the input text, specifically: The label features and level features of each level are respectively passed through a multi-layer perceptron to generate a label-based local prediction vector and a level-based global prediction vector, where the elements in the local prediction vector and the global prediction vector are the local prediction probability and the global prediction probability respectively; The confidence vector of the hierarchical prediction label is obtained by weighted averaging the elements at corresponding positions in the local prediction vector and the global prediction vector, where the elements in the confidence vector are the confidences of the hierarchical prediction labels of the input text by the fine-grained classification tree; The level prediction label corresponding to the highest confidence level above the threshold in the confidence vector is selected as the prediction label of the corresponding level; The predicted labels of each level are integrated as the predicted labels of the input text.

5. The method for detecting a large language model prompt word injection attack based on context learning according to claim 4 is characterized in that: In step 3, after the label-based attention module extracts the label features and hierarchical features of the hierarchical embedding vector, the hierarchical features of the previous layer need to be input into the label-based attention module of the next layer to extract the label features and hierarchical features together with the hierarchical embedding vector of the next layer itself.

6. The method for detecting a large language model prompt word injection attack based on context learning according to claim 1, characterized in that: In step 4, the context set and test prompt are constructed respectively based on the prediction output of the sampling sample and the test sample using the fine-tuned prediction model, specifically: Input the sample into the fine-tuned prediction model to obtain a first predicted label and a first confidence level. Based on the predicted label, confidence level, and label of the sample, an example template in the form of sample-predicted label-confidence level-true label is constructed and stored in the context set. The test sample is input into the fine-tuned prediction model to obtain the second prediction label and the second confidence. According to the example template, a test prompt of the form sample-prediction label-confidence is constructed.

7. The method for detecting a large language model prompt word injection attack based on contextual learning according to claim 6 is characterized in that: In step 4, the concatenated data is input into the large language model, and context learning is used to obtain the prediction results and cause analysis of the test samples, specifically: After concatenating the context set and the test prompt, they are input into the large language model together with the cause analysis prompt words; The large language model learns the example template corresponding to the context set, obtains the true label of the test sample based on the predicted label and confidence level corresponding to the test sample in the test prompt, and outputs the prediction result of the test sample in the form of the example template. The prediction result is the sample-predicted label-confidence-true label of the test sample; The large language model performs causal analysis on the reasoning process of the true label of the test sample based on the causal analysis prompt words.

8. A large language model prompt word injection attack detection device based on context learning, characterized in that: It includes an instruction data set construction unit, a fine-grained classification tree construction unit, a classification model construction unit, and a context prediction unit; The instruction data set construction unit is used to mutate the prompt word injection attack instruction based on the language mutation algorithm of fuzz testing to obtain the mutated attack instruction, perform a cyclic attack on the large language model through the mutated attack instruction, retain the mutated attack instruction that is effective, and construct the instruction data set; The fine-grained classification tree construction unit is used to design a multi-level and multi-label fine-grained classification tree, construct the first level of the fine-grained classification tree according to the attack purpose of the prompt word injection attack instruction, perform semantic division on the first level to obtain the second level, and obtain the third level according to the implementation strategy of each label in the second level; Use fine-grained classification trees to add prompt word injection attack category labels to the instruction dataset; The classification model construction unit is used to introduce a label-based attention module based on the BERT pre-training model and the fine-grained classification tree to construct a hierarchical multi-label classification model, wherein the fine-grained classification tree is used to stratify the input text and assign hierarchical prediction labels; the BERT pre-training model is used to obtain the hierarchical embedding vector of the hierarchical text; the label-based attention module is used to extract the label features and hierarchical features of the hierarchical embedding vector, and use the weighted average as the confidence of the hierarchical prediction label of the corresponding layer, and integrate the hierarchical prediction label corresponding to the highest confidence above the threshold as the prediction label of the input text; The context prediction unit is used to fine-tune the hierarchical multi-label classification model with a labeled instruction dataset to obtain a fine-tuned prediction model, obtain sampling samples of the labeled instruction dataset, and use the fine-tuned prediction model to construct a context set and test prompts for the prediction outputs of the sampling samples and test samples respectively. After splicing, they are input into the large language model, and context learning is used to obtain the prediction results and cause analysis of the test samples.

9. A large language model prompt word injection attack detection device based on context learning, comprising a memory and a processor, wherein the memory is used to store a computer program, characterized in that: The processor is configured to implement the context-learning-based large language model prompt word injection attack detection method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is used on a computer, the method for detecting a large language model prompt word injection attack based on context learning according to any one of claims 1 to 7 is implemented.

Images