A method, device, equipment and storage medium for detecting abnormality of prompt words

Abstract

The embodiments of the present application provide a method, apparatus, device, and storage medium for detecting anomalies in prompt words, which relate to the field of artificial intelligence technology. The method includes the following steps: obtaining a prompt word to be detected and the perplexity of the prompt word to be detected; if the perplexity exceeds a second threshold but does not exceed a first threshold, processing the prompt word to be detected in an adversarial perturbation manner to obtain a perturbed prompt word; obtaining the response semantics of the prompt word to be detected and the perturbation response semantics corresponding to the perturbation prompt word based on a large language model prediction; if the response semantics and the perturbation response semantics are inconsistent, the prompt word to be detected is abnormal. Since a dual threshold is set for perplexity detection, and after the prompt word to be detected is perturbated, the response semantics and the perturbation response semantics are compared to determine whether the prompt word is abnormal, the accuracy of prompt word anomaly detection can be effectively improved, and the security of the large language model can be enhanced.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and storage medium for detecting anomalies in prompt words. Background Art

[0002] What makes the AI ​​technology ChatGPT so amazing is that ChatGPT can combine context and answer questions logically like a human. Even if it generates extremely long texts, its answers will not deviate. With the popularization of ChatGPT, large language models (LLM, the full name of which is Large Language Model in English) have come into people's attention.

[0003] Large language models (LLMs) have massive parameters and are deep learning models trained on massive datasets to understand human language. However, when attackers directly or indirectly manipulate the trusted LLM through carefully crafted input prompts, the LLM ignores pre-set audit rules and executes malicious instructions, resulting in a prompt injection attack. These attacks have become a primary security risk for large language models.

[0004] Prior art uses malicious command detection methods to detect anomalies in prompt words. Specifically, based on specific scenarios and business needs, a blacklist of risky content is pre-established, including sensitive words, malicious commands, special characters, and so on. Anomalies in prompt words are then detected through keyword searches based on the blacklist. However, this method only filters keywords already on the pre-established blacklist, which has certain limitations. This results in low accuracy in detecting anomalies in prompt words, thus compromising the security of large language models. Summary of the Invention

[0005] The embodiments of the present application provide a method, apparatus, device, and storage medium for detecting anomalies in prompt words, which effectively improve the accuracy of detecting anomalies in prompt words and enhance the security of large language models.

[0006] In a first aspect, an embodiment of the present application provides a method for detecting anomalies in prompt words, comprising:

[0007] Obtaining a prompt word to be detected and a perplexity of the prompt word to be detected;

[0008] If the perplexity exceeds the second threshold and does not exceed the first threshold, the prompt word to be detected is processed in an adversarial perturbation manner to obtain a perturbation prompt word;

[0009] Obtaining the response semantics of the prompt word to be detected and the disturbance response semantics corresponding to the disturbance prompt word based on the large language model prediction;

[0010] If the response semantics are inconsistent with the disturbance response semantics, the prompt word to be detected is abnormal.

[0011] In an embodiment of the present application, by obtaining the prompt word to be detected and the perplexity of the prompt word to be detected, it is determined that the perplexity exceeds the second threshold but does not exceed the first threshold, and then the prompt word to be detected is processed in an adversarial perturbation manner to obtain the perturbation prompt word, and then the response semantics of the prompt word to be detected and the perturbation response semantics corresponding to the perturbation prompt word are predicted based on the large language model. If the response semantics and the perturbation response semantics are inconsistent, the prompt word to be detected is abnormal. Since the perplexity detection sets a double threshold, and after the prompt word to be detected is perturbated, the response semantics and the perturbation response semantics can be compared to determine whether the prompt word is abnormal, thereby effectively improving the accuracy of abnormality detection of the prompt word and enhancing the security of the large language model.

[0012] In an optional implementation manner, after obtaining the prompt word to be detected and the perplexity of the prompt word to be detected, the method further includes:

[0013] If the perplexity does not exceed a second threshold, determining the similarity between the prompt word to be detected and each reference prompt word in the vector database;

[0014] If the maximum similarity among the obtained multiple similarities exceeds the third threshold, the prompt word to be detected is abnormal.

[0015] In an optional implementation manner, the reference prompt words in the vector database are: machine-generated first-category prompt words, and / or manually designed second-category prompt words.

[0016] Under the above embodiment, by determining the similarity between the prompt word to be detected and each reference prompt word in the vector database, if the maximum similarity among the multiple similarities obtained exceeds the third threshold, the prompt word to be detected is abnormal, and the reference prompt words in the vector database are: machine-generated first-category prompt words, and / or manually designed second-category prompt words, which can effectively detect prompt words that can bypass the security protection measures of the large language model.

[0017] In an optional implementation, determining the similarity between the prompt word to be detected and each reference prompt word in the vector database includes:

[0018] Converting the prompt word to be detected into a prompt word vector to be detected through a text embedding model;

[0019] Based on the distance between the prompt word vector to be detected and the prompt word vector of each reference prompt word, the similarity between the prompt word to be detected and each reference prompt word is determined.

[0020] In an optional implementation, the temperature coefficient of the large language model does not exceed 0.6; the temperature coefficient is used to measure the stability of the output response of the large language model.

[0021] In the above embodiment, by setting the temperature coefficient of the large language model to a smaller value (not exceeding 0.6), the output of the large language model can be made relatively stable, the randomness of the output response can be reduced, and the accuracy of the judgment of the influence of the disturbance mode on the prompt word can be improved.

[0022] In an optional implementation, after determining that the prompt word to be detected is abnormal, the following operations are further included:

[0023] The prompt word to be detected is stored in the vector database as a second type of prompt word.

[0024] In the above implementation, after determining that the prompt word to be detected is abnormal, the prompt word to be detected is stored as a second type of prompt word in the vector database, which can effectively prevent similar prompt word attacks from occurring again in the future.

[0025] In an optional implementation, the adversarial perturbation methods include: synonym replacement, word order disruption, subword rearrangement, and morphological change.

[0026] In a second aspect, an embodiment of the present application provides a prompt word anomaly detection device, comprising:

[0027] An acquisition module, configured to acquire a prompt word to be detected and a perplexity of the prompt word to be detected;

[0028] a prompt word perturbation module, configured to process the prompt word to be detected using an adversarial perturbation method to obtain a perturbed prompt word if the perplexity exceeds the second threshold but does not exceed the first threshold;

[0029] a response comparison module, configured to obtain, based on a large language model, a response semantic of the prompt word to be detected and a disturbance response semantic corresponding to the disturbance prompt word;

[0030] The anomaly detection module is used to determine that the prompt word to be detected is abnormal if the response semantics are inconsistent with the disturbance response semantics.

[0031] In an optional implementation, a similarity comparison module is further included;

[0032] The similarity comparison module is specifically used for:

[0033] If the perplexity does not exceed a second threshold, determining the similarity between the prompt word to be detected and each reference prompt word in the vector database;

[0034] If the maximum similarity among the obtained multiple similarities exceeds the third threshold, the prompt word to be detected is abnormal.

[0035] In an optional implementation manner, the similarity comparison module is further configured to:

[0036] Converting the prompt word to be detected into a prompt word vector to be detected through a text embedding model;

[0037] Based on the distance between the prompt word vector to be detected and the prompt word vector of each reference prompt word, the similarity between the prompt word to be detected and each reference prompt word is determined.

[0038] In an optional embodiment, it further includes a prompt word storage module;

[0039] The prompt word storage module is specifically used for:

[0040] The prompt word to be detected is stored in the vector database as a second type of prompt word.

[0041] In a third aspect, an embodiment of the present application provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for detecting abnormalities in prompt words when executing the program.

[0042] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program executable by a computer device. When the program runs on the computer device, the computer device executes the steps of the above-mentioned prompt word anomaly detection method.

[0043] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned prompt word anomaly detection method. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 A schematic diagram of the structure of a system architecture provided in an embodiment of the present application;

[0046] Figure 2A flowchart of a method for detecting anomalies in prompt words provided in an embodiment of the present application;

[0047] Figure 3 A flowchart of a method for detecting anomalies in prompt words provided in an embodiment of the present application;

[0048] Figure 4 A schematic diagram of the structure of a prompt word anomaly detection device provided in an embodiment of the present application;

[0049] Figure 5 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and beneficial effects of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0051] In order to more clearly explain the embodiments of the present application, the following glossary is provided:

[0052] Temperature coefficient: controls the randomness and creativity of text generated by the language model and adjusts the probability of predicted words in the model output layer. The larger the temperature coefficient, the smaller the variance of the predicted word probability, that is, the probability of many words being selected increases, which is conducive to the diversification of text output.

[0053] Perplexity (PPL) In information theory, perplexity measures how well a probability distribution or probability model predicts a sample. It can also be used to compare two probability distributions or probability models. Generally speaking, a probability distribution or probability model with lower perplexity predicts a sample better.

[0054] Token: A computer term that means a token (temporary) in computer authentication.

[0055] GPT-2: (full name Generative Pre-Training 2.0) GPT-2 is the second-generation generative pre-training model launched by OpenAI (Open Artificial Intelligence) in 2019.

[0056] LightGBM: The full name is Light Gradient Boosting Machine. It is a machine learning algorithm based on gradient boosting tree. Compared with the traditional gradient boosting tree algorithm, LightGBM has greater advantages in training and prediction speed.

[0057] BPE-Dropout: BPE stands for Byte Pair Encoding. It decomposes words into character sequences and then iteratively combines the most frequently occurring character pairs into one. BPE-dropout is an effective BPE-based subword regularization method that can perform multiple segmentations on specific words.

[0058] Softmax layer: The softmax layer is the most common activation function in neural networks. It can convert the output of the neural network into a probability distribution. In deep learning, softmax is widely used in classification problems and some tasks that require output probability distribution.

[0059] Text adversarial attack: Text adversarial attack refers to making small perturbations to the input samples, causing the large model to make incorrect predictions about the perturbed samples.

[0060] GCG: Greedy Coordinate Gradient, greedy coordinate gradient, identifies a set of possible single-token replacements by utilizing token-level gradients, then evaluates the replacement loss of these candidates in the set and selects the one with the smallest replacement loss.

[0061] AutoDAN: Automatic adversarial attack, generates new tokens one by one (from left to right). When generating each token, it goes through two steps: preliminary selection and fine selection.

[0062] BERT: English full name: Bidirectional Encoder Representations from Transformers, BERT is a language representation model that aims to pre-train deep bidirectional representations by jointly regulating the left and right context in all layers.

[0063] See also Figure 1 , which is a system architecture diagram applicable to an embodiment of the present application. The system architecture includes at least a terminal device 101 and a detection system 102. The number of terminal devices 101 can be one or more, and the number of detection systems 102 can also be one or more. This application does not specifically limit the number of terminal devices 101 and detection systems 102.

[0064] Applications are pre-installed in the terminal device 101, where the applications are client applications, web applications, mini-program applications, etc. The terminal device 101 can be a smart phone, a tablet computer, a laptop computer, or a desktop computer, but is not limited thereto.

[0065] The detection system 102 is the background server of the application. The detection system 102 can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal device 101 and the detection system 102 can be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0066] The method for detecting abnormalities in prompt words in the embodiment of the present application may be executed by the terminal device 101, or by the detection system 102, or may be executed interactively by the terminal device 101 and the detection system 102.

[0067] based on Figure 1 The system architecture diagram shown in FIG. 1 is a flowchart of a method for detecting abnormalities of a prompt word provided by an embodiment of the present application. Figure 2 As shown, the process of the method is executed by a computer device, which may be Figure 1 The terminal device 101 and / or detection system 102 shown includes the following steps:

[0068] Step 201: Obtain a prompt word to be detected and the perplexity of the prompt word to be detected.

[0069] Specifically, we obtain the prompt word to be detected and calculate the perplexity of the prompt word to be detected. The perplexity is the exponential average of the negative log-likelihood function of the sequence. If the sentence is not fluent, contains grammatical errors, or is inconsistent with the previous text, the perplexity value will increase. The specific calculation method is shown in formula (1):

[0070]

[0071] Where X is a token sequence of length t (x1, x2, ..., x t ).

[0072] Generally, GPT-2 or BERT is used to calculate the sequence perplexity. If the input sequence exceeds the longest token limit of GPT-2 (assuming it is k), a sliding window (the window length is k) is used to calculate the perplexity.

[0073] In addition, there are two options for constructing a classifier for anomaly detection of prompt words: one is to construct it based on a single factor of perplexity, and the other is to comprehensively consider perplexity and sequence length. However, if the classifier is constructed based on perplexity alone, the reference threshold of perplexity is 997. Setting a single perplexity threshold will lead to problems with high false alarm rates or missed alarm rates. If perplexity and sequence length are comprehensively considered, the classifier can be trained using, but not limited to, the LightGBM algorithm, and this application does not impose any restrictions on this.

[0074] Considering that constructing a classifier based on only the perplexity factor, setting a single threshold may lead to the problem of too high false alarm rate or missed alarm rate, the embodiment of the present application sets two thresholds when using perplexity for binary classification.

[0075] Step 202: If the perplexity exceeds the second threshold but does not exceed the first threshold, an adversarial perturbation method is used to process the prompt word to be detected to obtain a perturbed prompt word.

[0076] In an optional implementation, the adversarial perturbation methods include: synonym replacement, word order disruption, subword rearrangement, and morphological change.

[0077] Specifically, the second threshold is smaller than the first threshold. Assuming that the first threshold is set to 1000 and the second threshold is set to 400, if the perplexity exceeds the first threshold, the prompt word to be detected is directly considered abnormal, for example, the prompt word to be detected contains adversarial text.

[0078] If the perplexity exceeds the second threshold but does not exceed the first threshold, it is considered that the prompt word to be detected may be abnormal. Further, an adversarial perturbation method is used to process the prompt word to be detected to obtain a perturbed prompt word.

[0079] First, the prompt word to be detected is normalized to obtain the normalized prompt word to be detected. Taking the input string prompt as an example, the normalization process of the input string prompt is as follows:

[0080] 1) Convert the input string to lowercase letters;

[0081] 2) Delete extra spaces (including spaces at the beginning and end of the string, replace multiple consecutive spaces with a single space, and delete spaces between special characters);

[0082] 3) Of multiple consecutive special characters, only the first one is retained;

[0083] Then, the standardized prompt words to be detected are randomly processed to obtain disturbed prompt words. The random processing methods include but are not limited to the BPE-Dropout method. The embodiment of the present application takes the BPE-Dropout method as an example, selects a BPE-D value of 0.4, randomly discards p% of the BPE and merges them, generates random word segmentations, and obtains the disturbed prompt word prompt_disturbed corresponding to the input string prompt.

[0084] In an optional implementation, if the perplexity does not exceed the second threshold, the similarity between the prompt word to be detected and each reference prompt word in the vector database is determined; if the maximum similarity among the multiple similarities obtained exceeds the third threshold, the prompt word to be detected is abnormal.

[0085] In an optional implementation, the reference prompt words in the vector database are: machine-generated first-category prompt words, and / or manually designed second-category prompt words.

[0086] Specifically, artificially designed prompt word injection attacks are usually semantically smooth, making it difficult to directly detect prompt word anomalies through perplexity. Therefore, the prompt words to be detected are further detected through similarity retrieval methods. Similarity retrieval methods include: vector database comparison method, Approximate Nearest Neighbor (ANN), Locality-Sensitive Hashing (LSH), Product Quantization (PQ), and Hierarchical Navigable Small World (HNSW). The embodiment of this application uses the vector database comparison method as an example to illustrate. Mainstream vector databases include pinecone, milvus, annoy, etc.

[0087] The construction process of the vector database is as follows:

[0088] Step 1: Prepare the initial corpus.

[0089] Specifically, the initial corpus contains two categories: harmful behaviors and instructions, namely the first category of machine-generated prompt words; and the second category of manually designed prompt words. Among them, harmful behaviors and instructions include but are not limited to various questions and instructions in the AdvBench and Safety-Prompts data sets. The second category of manually designed prompt words refers to manually designed prompt words used to bypass the security protection measures of the large language model, that is, by designing prompt words to remove the built-in security restrictions of the large language model, thereby guiding or controlling the large language model to generate non-compliant or harmful content that should have been blocked.

[0090] Step 2: Text vectorization.

[0091] Specifically, a text embedding model is used to convert each reference prompt word in the initial corpus into a fixed-length vector.

[0092] If the perplexity does not exceed the second threshold, the similarity between the detected prompt word and each reference prompt word in the vector database is determined. There are two vector databases, each with a corresponding similarity threshold. The reference prompt words in the first vector database are machine-generated first-category prompt words; the reference prompt words in the second vector database are manually designed second-category prompt words. A vector database is selected as needed to determine the similarity between the detected prompt word and each reference prompt word in the vector database. For example, Option 1: Select the first vector database and determine the similarity between the detected prompt word and each reference prompt word in the first vector database. Option 2: Select the second vector database and determine the similarity between the detected prompt word and each reference prompt word in the second vector database. Option 3: Select the first vector database and the second vector database and determine the similarity between the detected prompt word and each reference prompt word in the first and second vector databases. The maximum similarity among the multiple similarities is obtained. If the maximum similarity exceeds the third threshold, the detected prompt word is considered abnormal.

[0093] As another implementation, the number of times that the multiple similarities exceed the third threshold is counted. If the number of times that the multiple similarities exceed the third threshold meets a preset standard, the prompt word to be detected is abnormal.

[0094] Under the above embodiment, by determining the similarity between the prompt word to be detected and each reference prompt word in the vector database, if the maximum similarity among the multiple similarities obtained exceeds the third threshold, the prompt word to be detected is abnormal, and the reference prompt word in the vector database is a machine-generated first-category prompt word, and / or a manually designed second-category prompt word. Because the vector database has a special vector retrieval and query function, existing or possible abnormal prompt words are embedded and stored in the vector database, which can not only effectively prevent similar attacks, but also effectively detect prompt words that can bypass the security protection measures of the large language model.

[0095] In an optional implementation, the prompt word to be detected is converted into a prompt word vector to be detected through a text embedding model; and based on the distance between the prompt word vector to be detected and the prompt word vector of each reference prompt word, the similarity between the prompt word to be detected and each reference prompt word is determined.

[0096] Specifically, text embedding models include Word2Vec (word to vector), BERT (Bidirectional Encoder Representations from Transformers), and FastText (a shallow network model). Similarity metrics include Euclidean distance, Hamming distance, cosine similarity distance, and Manhattan distance. This application example uses cosine similarity distance as an example.

[0097] The prompt word to be detected is converted into a prompt word vector to be detected through the text embedding model. The cosine similarity distance between the prompt word vector to be detected and the prompt word vector of each reference prompt word is calculated. The similarity between the prompt word to be detected and each reference prompt word is determined based on the cosine similarity distance. The calculation formula of the cosine similarity distance is shown in formula (2):

[0098]

[0099] Among them, A and B are the vectors of two reference prompt words respectively, and θ is the angle between the two reference prompt words.

[0100] Step 203 : obtaining the response semantics of the prompt word to be detected and the disturbance response semantics corresponding to the disturbance prompt word based on the large language model prediction.

[0101] Specifically, large language models typically begin questions they can answer with affirmative sentences like "Sure, here's" or "Of course!", while unanswerable questions typically begin with negative sentences like "I'm sorry" or "As an AI assistant." Therefore, the effectiveness of a perturbation—that is, whether the prompt word is abnormal—can be determined by comparing the large language model's responses to the prompt word before and after the perturbation.

[0102] The detection prompt and the disturbance prompt are fed into the large language model to obtain the response semantics of the detection prompt and the disturbance response semantics corresponding to the disturbance prompt. For example, the detection prompt word prompt and the disturbance prompt word prompt_disturbed are fed into the large language model to obtain the response semantics response and the disturbance response semantics response_disturbed.

[0103] In an optional implementation, a temperature coefficient of the large language model does not exceed 0.6; the temperature coefficient is used to measure the stability of an output response of the large language model.

[0104] Specifically, the large language model is the prediction of the next word, and its output is the probability distribution p of the word i, in the final softmax layer, the probability distribution of words p i See calculation formula (3):

[0105]

[0106] Among them, the temperature coefficient T can be x i (logits) adjusted to x i Because the temperature coefficient can affect the diversity of the large language model's output, the lower the temperature coefficient value, the more stable the model output, and the higher the temperature coefficient value, the greater the diversity of the model output. In order to reduce the randomness of the large language model's output and effectively determine the impact of disturbance processing on the prompt word to be detected, the temperature coefficient of the large language model in the embodiment of the present application is set to no more than 0.6.

[0107] In the above embodiment, by setting the temperature coefficient of the large language model to a smaller value (not exceeding 0.6), the output of the large language model can be made relatively stable, the randomness of the output response can be reduced, and the accuracy of the judgment of the influence of the disturbance mode on the prompt word can be improved.

[0108] Step 204: If the response semantics are inconsistent with the disturbance response semantics, the prompt word to be detected is abnormal.

[0109] In an embodiment of the present application, by obtaining the prompt word to be detected and the perplexity of the prompt word to be detected, it is determined that the perplexity exceeds the second threshold but does not exceed the first threshold, and then the prompt word to be detected is processed in an adversarial perturbation manner to obtain the perturbation prompt word, and then the response semantics of the prompt word to be detected and the perturbation response semantics corresponding to the perturbation prompt word are predicted based on the large language model. If the response semantics and the perturbation response semantics are inconsistent, the prompt word to be detected is abnormal. Since the perplexity detection sets a double threshold, and after the prompt word to be detected is perturbated, the response semantics and the perturbation response semantics can be compared to determine whether the prompt word is abnormal, thereby effectively improving the accuracy of abnormality detection of the prompt word and enhancing the security of the large language model.

[0110] In an optional implementation, a vector database for affirmative sentences and a vector database for negative sentences are constructed, and semantic analysis is performed on the response semantics and the disturbance response semantics based on the vector databases. If the response semantics and the disturbance response semantics are affirmative sentence responses and / or negative sentence responses, respectively, the response semantics and the disturbance response semantics are inconsistent, and the prompt word to be detected is abnormal.

[0111] Under the above implementation mode, by processing the prompt word to be detected in an adversarial perturbation manner and performing semantic comparative analysis on the response semantics and perturbation response semantics before and after the prompt word to be detected is disturbed, it is possible to re-determine whether the prompt word to be detected has an abnormality, which is conducive to improving the accuracy of abnormality detection of the prompt word.

[0112] In an optional implementation, after determining that the prompt word to be detected is abnormal, the prompt word to be detected is saved in the vector database as a second type of prompt word.

[0113] Specifically, the abnormal prompt words to be detected include: the prompt words to be detected contain adversarial text, the prompt words to be detected contain malicious instructions, and the prompt words to be detected contain artificially designed second-category prompt words.

[0114] In the above implementation, after determining that the prompt word to be detected is abnormal, the prompt word to be detected is stored as a second type of prompt word in the vector database, which can effectively prevent similar prompt word attacks from occurring again in the future.

[0115] To better explain the embodiments of this application, see Figure 3 , the embodiment of the present application provides a flowchart of a method for detecting anomalies of prompt words, which specifically includes the following contents:

[0116] The cue word to be detected is input into the perplexity detection module to obtain the perplexity of the cue word to be detected. If the perplexity exceeds a first threshold, the cue word to be detected is abnormal and is added to the vector database. If the perplexity does not exceed the first threshold but exceeds a second threshold, the cue word to be detected is perturbed using an adversarial perturbation method to obtain a perturbed cue word. The cue word to be detected and the perturbed cue word are respectively input into the large language model to obtain the response semantics of the cue word to be detected and the perturbation response semantics corresponding to the perturbation cue word.

[0117] Analyze and compare whether the response semantics and the perturbation response semantics are consistent. If the response semantics and the perturbation response semantics are inconsistent, the prompt word to be detected is abnormal and the prompt word to be detected is added to the vector database; if the response semantics and the perturbation response semantics are consistent, the prompt word to be detected is normal.

[0118] If the perplexity does not exceed the second threshold, the similarity between the prompt word to be detected and each reference prompt word in the vector database is determined. If the maximum similarity among the multiple similarities obtained exceeds the third threshold, the prompt word to be detected is abnormal and the prompt word to be detected is added to the vector database; if the maximum similarity among the multiple similarities does not exceed the third threshold, the prompt word to be detected is normal.

[0119] In addition, the embodiments of the present application are explained using a text anti-attack method as an example.

[0120] For example, if the perplexity of the prompt word to be detected input by the GCG method exceeds the threshold 1 through perplexity detection, the prompt word to be detected is directly considered to be abnormal; if the perplexity of the prompt word to be detected input by the AutoDAN method does not exceed the threshold 1 but exceeds the threshold 2 through perplexity detection, it enters the prompt word perturbation processing process. During the input normalization processing, some special characters (such as "\") are deleted, and then the standardized prompt word to be detected is randomly perturbed to obtain a perturbed prompt word. The large language model makes positive and negative semantic responses to the prompt word to be detected and the perturbed prompt word, respectively. In this case, the prompt word to be detected input by the AutoDAN method is abnormal and is saved in the vector database.

[0121] Based on the same technical concept, see Figure 4 , an embodiment of the present application provides a prompt word anomaly detection device, comprising:

[0122] An acquisition module 401 is used to acquire a prompt word to be detected and a perplexity of the prompt word to be detected;

[0123] The prompt word perturbation module 402 is configured to process the prompt word to be detected using an adversarial perturbation method to obtain a perturbed prompt word if the perplexity exceeds the second threshold but does not exceed the first threshold;

[0124] A response comparison module 403 is configured to obtain the response semantics of the prompt word to be detected and the disturbance response semantics corresponding to the disturbance prompt word based on the prediction of the large language model;

[0125] The anomaly detection module 405 is configured to determine that the prompt word to be detected is abnormal if the response semantics are inconsistent with the perturbation response semantics.

[0126] In an optional embodiment, a similarity comparison module 404 is further included;

[0127] The similarity comparison module 404 is specifically used for:

[0128] If the perplexity does not exceed the second threshold, determining the similarity between the prompt word to be detected and each reference prompt word in the vector database;

[0129] If the maximum similarity among the obtained multiple similarities exceeds the third threshold, the prompt word to be detected is abnormal.

[0130] In an optional implementation, the similarity comparison module 404 is further configured to:

[0131] Convert the prompt word to be detected into a prompt word vector through the text embedding model;

[0132] Based on the distance between the prompt word vector to be detected and the prompt word vector of each reference prompt word, the similarity between the prompt word to be detected and each reference prompt word is determined.

[0133] In an optional embodiment, a prompt word storage module 406 is further included;

[0134] The prompt word storage module 406 is specifically used for:

[0135] The prompt words to be detected are saved in the vector database as the second type of prompt words.

[0136] In an embodiment of the present application, by obtaining the prompt word to be detected and the perplexity of the prompt word to be detected, it is determined that the perplexity exceeds the second threshold but does not exceed the first threshold, and then the prompt word to be detected is processed in an adversarial perturbation manner to obtain the perturbation prompt word, and then the response semantics of the prompt word to be detected and the perturbation response semantics corresponding to the perturbation prompt word are predicted based on the large language model. If the response semantics and the perturbation response semantics are inconsistent, the prompt word to be detected is abnormal. Since the perplexity detection sets a double threshold, and after the prompt word to be detected is perturbated, the response semantics and the perturbation response semantics can be compared to determine whether the prompt word is abnormal, thereby effectively improving the accuracy of abnormality detection of the prompt word and enhancing the security of the large language model.

[0137] Based on the same technical concept, the embodiment of the present application provides a computer device, which can be Figure 1 The terminal equipment and / or detection system shown, such as Figure 5 As shown, it includes at least one processor 501 and a memory 502 connected to the at least one processor. The specific connection medium between the processor 501 and the memory 502 is not limited in the embodiment of the present application. Figure 5 For example, the processor 501 and the memory 502 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, and the like.

[0138] In the embodiment of the present application, the memory 502 stores instructions that can be executed by at least one processor 501. The at least one processor 501 can perform the steps of the above-mentioned prompt word abnormality detection method by executing the instructions stored in the memory 502.

[0139] The processor 501 is the control center of the computer device. It can connect the various parts of the computer device using various interfaces and lines. By running or executing instructions stored in the memory 502 and calling data stored in the memory 502, it can realize abnormal detection of prompt words. Optionally, the processor 501 may include one or more processing units. The processor 501 may integrate an application processor and a modem processor. The application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 501. In some embodiments, the processor 501 and the memory 502 may be implemented on the same chip. In some embodiments, they may also be implemented on separate chips.

[0140] The processor 501 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0141] The memory 502 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 502 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 502 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer device, but is not limited thereto. The memory 502 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0142] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program that can be executed by a computer device. When the program runs on the computer device, the computer device executes the steps of the above-mentioned prompt word anomaly detection method.

[0143] Based on the same inventive concept, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned prompt word anomaly detection method.

[0144] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0145] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as a combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer device or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0146] These computer program instructions may also be stored in a computer readable memory that can direct a computer device or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0147] These computer program instructions can also be loaded onto a computer device or other programmable data processing device so that a series of operating steps are executed on the computer device or other programmable device to produce a process implemented by the computer device, thereby providing instructions for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0148] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0149] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A method for detecting abnormalities in prompt words, characterized in that: include: Obtaining a prompt word to be detected and the perplexity of the prompt word to be detected, where the perplexity is the exponential average of the negative log-likelihood function of the prompt word to be detected; If the perplexity exceeds the second threshold and does not exceed the first threshold, the prompt word to be detected is processed in an adversarial perturbation manner to obtain a perturbation prompt word; Obtaining, based on the large language model prediction, the response semantics of the prompt word to be detected and the disturbance response semantics corresponding to the disturbance prompt word; the response semantics is an affirmative sentence response or a negative sentence response; the disturbance response semantics is an affirmative sentence response or a negative sentence response; If the response semantics are inconsistent with the disturbance response semantics, the prompt word to be detected is abnormal.

2. The method according to claim 1, wherein After obtaining the prompt word to be detected and the perplexity of the prompt word to be detected, the method further includes: If the perplexity does not exceed a second threshold, determining the similarity between the prompt word to be detected and each reference prompt word in the vector database; If the maximum similarity among the obtained multiple similarities exceeds the third threshold, the prompt word to be detected is abnormal.

3. The method according to claim 2, wherein The reference prompt words in the vector database are: machine-generated first-category prompt words, and / or manually designed second-category prompt words.

4. The method according to claim 2, wherein Determining the similarity between the prompt word to be detected and each reference prompt word in the vector database includes: Converting the prompt word to be detected into a prompt word vector to be detected through a text embedding model; Based on the distance between the prompt word vector to be detected and the prompt word vector of each reference prompt word, the similarity between the prompt word to be detected and each reference prompt word is determined.

5. The method according to claim 1, wherein The temperature coefficient of the large language model does not exceed 0.6; the temperature coefficient is used to measure the stability of the output response of the large language model.

6. The method according to claim 3, wherein After determining that the prompt word to be detected is abnormal, the following operations are also included: The prompt word to be detected is stored in the vector database as a second type of prompt word.

7. The method according to any one of claims 1 to 6, characterized in that: The adversarial perturbation methods include: synonym replacement, word order disruption, subword rearrangement, and morphological change.

8. A device for detecting abnormalities in prompt words, characterized in that: include: An acquisition module, configured to acquire a prompt word to be detected and a perplexity of the prompt word to be detected, wherein the perplexity is an exponential average of a negative log-likelihood function of the prompt word to be detected; a prompt word perturbation module, configured to process the prompt word to be detected using an adversarial perturbation method to obtain a perturbed prompt word if the perplexity exceeds the second threshold but does not exceed the first threshold; a response comparison module, configured to predict, based on a large language model, the response semantics of the prompt word to be detected and the disturbance response semantics corresponding to the disturbance prompt word; the response semantics being an affirmative sentence response or a negative sentence response; and the disturbance response semantics being an affirmative sentence response or a negative sentence response; The anomaly detection module is used to determine that the prompt word to be detected is abnormal if the response semantics are inconsistent with the disturbance response semantics.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that It stores a computer program that can be executed by a computer device. When the program is run on the computer device, the computer device executes the steps of any one of the methods according to claims 1 to 7.

11. A computer program product, characterized in that The computer program product comprises a computer program stored on a computer-readable storage medium, wherein the computer program comprises program instructions. When the program instructions are executed by a computer device, the computer device is caused to perform the steps of the method according to any one of claims 1 to 7.

Images