Prompt anomaly detection method and apparatus, and device and storage medium

Abstract

The embodiments of the present application relate to the technical field of artificial intelligence. Provided are a prompt anomaly detection method and apparatus, and a device and a storage medium. The method comprises: acquiring a prompt to be subjected to detection and the perplexity of said prompt; if the perplexity exceeds a second threshold value and does not exceed a first threshold value, using an adversarial perturbation mode to process said prompt, in order to acquire a perturbed prompt; on the basis of prediction by a large language model, obtaining response semantics of said prompt and perturbed response semantics corresponding to the perturbed prompt; and if the response semantics and the perturbed response semantics are inconsistent, said prompt being anomalous. Two threshold values are set for perplexity detection, and after perturbation processing is performed on a prompt to be subjected to detection, whether the prompt is anomalous can be determined again by means of the comparison between response semantics and perturbed response semantics, so that the accuracy of prompt anomaly detection can be effectively improved, thereby enhancing the security of large language models.

Description

Anomaly detection method, device and equipment for prompt words and storage medium

[0001] Cross-reference to related applications

[0002] The present application claims priority to the Chinese patent application No. 202410437421.4, filed on April 11, 2024, entitled "Anomaly detection method, device and equipment for prompt words and storage medium", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0003] Embodiments of the present application relate to the field of artificial intelligence technology, in particular to an anomaly detection method, device and equipment for prompt words and storage medium. BACKGROUND

[0004] The reason why AI technology ChatGPT is amazing is that ChatGPT can combine context and answer questions logically like a person, even if it generates a long text, the answer will not deviate. With the popularity of ChatGPT, large language models (LLM, English full name Large Language Model) have entered people's eyes.

[0005] Large language models have large-scale parameters, which are deep learning models trained on large data sets to understand human language. However, when an attacker directly or indirectly manipulates a trusted LLM through a carefully designed input prompt, the LLM will ignore the preset review rules and execute malicious instructions, i.e. be subjected to prompt injection attacks. Prompt injection attacks have become the primary risk to the security of large language models.

[0006] In the prior art, an anomaly detection method for prompt words is used. Specifically, according to the scene and business requirements, a blacklist with risky content is prepared in advance, which includes sensitive words, malicious instructions, special characters, etc. Then, based on the blacklist, the anomaly detection of the prompt words is performed through keyword retrieval. However, the anomaly detection method for prompt words only has filtering effect on the existing keywords in the pre-prepared blacklist, which has certain limitations, resulting in low accuracy of the anomaly detection of the prompt words, thereby affecting the security of the large language model. SUMMARY

[0007] Embodiments of the present application provide an anomaly detection method, device and equipment for prompt words and storage medium, which effectively improve the accuracy of the anomaly detection of the prompt words and enhance the security of the large language model.

[0008] In a first aspect, the embodiments of the present application provide an anomaly detection method for prompt words, comprising:

[0009] obtain a perplexity of the to-be-detected prompt word;

[0010] if the perplexity exceeds the second threshold value and does not exceed the first threshold value, the to-be-detected prompt word is processed in an adversarial perturbation manner to obtain a perturbed prompt word;

[0011] obtain a response semantic of the to-be-detected prompt word and a perturbed response semantic corresponding to the perturbed prompt word based on the large language model prediction;

[0012] if the response semantic and the perturbed response semantic are inconsistent, the to-be-detected prompt word is abnormal.

[0013] In the embodiments of the present application, by obtaining the to-be-detected prompt word and the perplexity of the to-be-detected prompt word, it is determined that the perplexity exceeds the second threshold value and does not exceed the first threshold value, then the to-be-detected prompt word is processed in an adversarial perturbation manner to obtain a perturbed prompt word, and then a response semantic of the to-be-detected prompt word and a perturbed response semantic corresponding to the perturbed prompt word are obtained based on the large language model prediction. If the response semantic and the perturbed response semantic are inconsistent, the to-be-detected prompt word is abnormal. Since the perplexity detection sets double thresholds, and after the perturbation processing of the to-be-detected prompt word, the comparison of the response semantic and the perturbed response semantic can determine whether the prompt word is abnormal again, thereby effectively improving the accuracy of the abnormal detection of the prompt word and enhancing the security of the large language model.

[0014] In an optional implementation, after the to-be-detected prompt word and the perplexity of the to-be-detected prompt word are obtained, the method further includes:

[0015] if the perplexity does not exceed the second threshold value, determining a similarity between the to-be-detected prompt word and each reference prompt word in the vector database;

[0016] if a maximum similarity in the obtained multiple similarities exceeds a third threshold value, the to-be-detected prompt word is abnormal.

[0017] In an optional implementation, the reference prompt words in the vector database are: machine-generated first-type prompt words, and / or artificially designed second-type prompt words.

[0018] In the above implementation, by determining the similarity between the to-be-detected prompt word and each reference prompt word in the vector database, if a maximum similarity in the obtained multiple similarities exceeds a third threshold value, the to-be-detected prompt word is abnormal. Moreover, the reference prompt words in the vector database are: machine-generated first-type prompt words, and / or artificially designed second-type prompt words, which can effectively detect prompt words that can bypass the security measures of the large language model.

[0019] In an optional implementation, the determining the similarity between the to-be-detected prompt and each reference prompt in the vector database comprises:

[0020] converting the to-be-detected prompt into a to-be-detected prompt vector through a text embedding model;

[0021] determining the similarity between the to-be-detected prompt and each reference prompt based on the distance between the to-be-detected prompt vector and the prompt vector of each reference prompt.

[0022] In an optional implementation, the temperature coefficient of the large language model is not more than 0.6; the temperature coefficient is used to measure the stability of the response output by the large language model.

[0023] In the above implementation, by setting the temperature coefficient of the large language model to a small value (not more than 0.6), the large language model can output relatively stably, the randomness of the response output is reduced, and the accuracy of the influence judgment of the perturbation mode on the prompt is improved.

[0024] In an optional implementation, after determining that the to-be-detected prompt is abnormal, the following operations are further included:

[0025] saving the to-be-detected prompt as a second type of prompt in the vector database.

[0026] In the above implementation, after determining that the to-be-detected prompt is abnormal, the to-be-detected prompt is saved as a second type of prompt in the vector database, which can effectively prevent similar prompt attacks from occurring again in the future.

[0027] In an optional implementation, the adversarial perturbation mode comprises synonym replacement, word sequence scrambling, subword rearrangement, and morphological change.

[0028] In a second aspect, an embodiment of the present application provides a prompt abnormality detection device, comprising:

[0029] an acquisition module configured to acquire a to-be-detected prompt and a perplexity of the to-be-detected prompt;

[0030] a prompt perturbation module configured to, if the perplexity exceeds a second threshold value and does not exceed a first threshold value, process the to-be-detected prompt using an adversarial perturbation mode to obtain a perturbed prompt;

[0031] a response comparison module configured to obtain a response semantic of the to-be-detected prompt and a perturbed response semantic corresponding to the perturbed prompt based on a large language model prediction;

[0032] an abnormality detection module configured to, if the response semantic and the perturbed response semantic are inconsistent, determine that the to-be-detected prompt is abnormal.

[0033] In an optional implementation, the method further includes a similarity comparison module.

[0034] The similarity comparison module is specifically configured to:

[0035] If the perplexity does not exceed the second threshold, determine the similarity between the to-be-detected trigger word and each reference trigger word in the vector database.

[0036] If the maximum similarity in the obtained plurality of similarities exceeds a third threshold, the to-be-detected trigger word is abnormal.

[0037] In an optional implementation, the similarity comparison module is further configured to:

[0038] convert the to-be-detected trigger word into a to-be-detected trigger word vector by using a text embedding model;

[0039] determine the similarity between the to-be-detected trigger word and each reference trigger word based on the distance between the to-be-detected trigger word vector and the trigger word vector of each reference trigger word.

[0040] In an optional implementation, the method further includes a trigger word storage module.

[0041] The trigger word storage module is specifically configured to:

[0042] save the to-be-detected trigger word as a second type of trigger word in the vector database.

[0043] In a third aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the above trigger word abnormality detection method when executing the program.

[0044] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program executable by a computer device, and when the program is executed on the computer device, the computer device executes the steps of the above trigger word abnormality detection method.

[0045] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program stored in a computer readable storage medium, and the computer program includes program instructions, and when the program instructions are executed by a computer device, the computer device executes the steps of the above trigger word abnormality detection method. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to make the technical solutions in the embodiments of the present application clearer, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0047] Fig. 1 is a structural schematic diagram of a system architecture provided by an embodiment of the present application;

[0048] Fig. 2 is a flow schematic diagram of an abnormality detection method of a prompt word provided by an embodiment of the present application;

[0049] Fig. 3 is a flow schematic diagram of an abnormality detection method of a prompt word provided by an embodiment of the present application;

[0050] Fig. 4 is a structural schematic diagram of an abnormality detection device of a prompt word provided by an embodiment of the present application;

[0051] Fig. 5 is a structural schematic diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purposes, technical solutions and beneficial effects of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0053] In order to more clearly explain the embodiments of the present application, the following terms are explained:

[0054] Temperature coefficient: controls the randomness and creativity of the language model to generate text, adjusts the probability of the predicted word of the model output layer, the greater the temperature coefficient, the smaller the variance of the probability of the predicted word, that is, the probability of selecting many words increases, which is beneficial to the diversification of text output.

[0055] Perplexity: (perplexity, PPL for short) In information theory, perplexity is used to measure the goodness of a probability distribution or a probability model in predicting samples. It can also be used to compare two probability distributions or probability models. Generally speaking, a probability distribution model or a probability model with low perplexity can better predict samples.

[0056] Token: computer term, meaning token (temporary) in computer identity authentication.

[0057] GPT-2: (full name Generative Pre-Training 2.0) GPT-2 is the second generation generative pre-training model launched by OpenAI (Open Artificial Intelligence) in 2019.

[0058] LightGBM: Light Gradient Boosting Machine, is a machine learning algorithm based on gradient boosting trees. Compared with traditional gradient boosting tree algorithm, LightGBM has greater advantages in training and prediction speed.

[0059] BPE-Dropout: BPE stands for Byte Pair Encoding, which decomposes words into character sequences and then iteratively combines the most frequent character pairs into one. BPE-dropout is an effective subword regularization method based on BPE, which can perform multiple segmentations on specific words.

[0060] Softmax layer: Softmax layer is the most common activation function in neural networks, which can convert the output of neural networks into probability distribution. In deep learning, softmax is widely used in classification problems and some tasks that require output probability distribution.

[0061] Textual adversarial attack: Textual adversarial attack refers to making small perturbations to input samples to make large models produce incorrect predictions on perturbed samples.

[0062] GCG: Greedy Coordinate Gradient, which uses token-level gradients to identify a set of possible single-token replacements, then evaluates the replacement loss of these candidates in the set, and selects the one with the smallest replacement loss.

[0063] AutoDAN: Automatic adversarial attack, which generates new tokens one by one (from left to right). When generating each token, it goes through two steps, Preliminary selection and Fine Selection, two inner loops.

[0064] BERT: English full name: Bidirectional Encoder Representations from Transformers, BERT is a language representation model that aims to pre-train deep bidirectional representations by jointly conditioning on all layers of left and right context.

[0065] Referring to FIG. 1, it is a system architecture diagram suitable for embodiments of the present application. The system architecture includes at least a terminal device 101 and a detection system 102. The number of terminal devices 101 can be one or more, and the number of detection systems 102 can also be one or more. The present application does not specifically limit the number of terminal devices 101 and detection systems 102.

[0066] The terminal device 101 pre-installs an application, where the application is a client application, a web application, a mini-program application, etc. The terminal device 101 can be a smart phone, a tablet computer, a notebook computer, or a desktop computer, but is not limited thereto.

[0067] The detection system 102 is a background server of the application, and the detection system 102 can be a standalone physical server, a server cluster composed of multiple physical servers, or a distributed system, and can also be a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal device 101 and the detection system 102 can be directly or indirectly connected through wired or wireless communication, which is not limited in the present application.

[0068] The prompt word anomaly detection method in the embodiments of the present application can be executed by the terminal device 101 or the detection system 102, or can be executed by the terminal device 101 and the detection system 102 in interaction.

[0069] Based on the system architecture diagram shown in FIG. 1, the present application embodiment provides a flowchart of a prompt word anomaly detection method, as shown in FIG. 2, the flow of the method is executed by a computer device, which can be the terminal device 101 and / or the detection system 102 shown in FIG. 1, including the following steps:

[0070] Step 201, obtaining a to-be-detected prompt word and a perplexity of the to-be-detected prompt word.

[0071] Specifically, the to-be-detected prompt word is obtained, and the perplexity of the to-be-detected prompt word is calculated. The perplexity is the exponential average of the negative log-likelihood function of the sequence. If the sentence is not smooth, contains syntax errors, or is inconsistent with the previous text in logic, the perplexity value will increase. The specific calculation method is shown in formula (1):

[0072] Where X is a token sequence (x1, x2,..., x t ).

[0073] Generally, GPT-2 or BERT is used to calculate the sequence perplexity. If the input sequence exceeds the maximum token limit of GPT-2 (assuming k), a sliding window (window length k) is used to calculate the perplexity.

[0074] In addition, there are two schemes for constructing a classifier for anomaly detection of the prompt word, one is to construct only according to the perplexity single factor, and the other is to comprehensively consider the perplexity and sequence length. However, if the classifier is constructed only according to the perplexity single factor, the reference threshold of the perplexity is 997, and the setting of the single threshold of the perplexity will cause the problem of high false alarm rate or missed alarm rate. If the perplexity and the sequence length are comprehensively considered, the classifier can be trained using but not limited to the LightGBM algorithm, which is not limited in the present application.

[0075] Considering that the setting of the single threshold of the classifier constructed only according to the perplexity single factor will cause the problem of high false alarm rate or missed alarm rate, two thresholds are set when the perplexity is used for binary classification in the embodiments of the present application.

[0076] In step 202, if the perplexity exceeds the second threshold and does not exceed the first threshold, the adversarial perturbation method is used to process the prompt word to be detected to obtain a perturbed prompt word.

[0077] In an optional embodiment, the adversarial perturbation method includes synonym replacement, word sequence disorder, subword rearrangement, and morphological change.

[0078] Specifically, the second threshold is less than the first threshold. Assuming that the first threshold is set to 1000 and the second threshold is set to 400, if the perplexity exceeds the first threshold, it is directly considered that the prompt word to be detected is abnormal, for example, the prompt word to be detected contains adversarial text.

[0079] If the perplexity exceeds the second threshold and does not exceed the first threshold, it is considered that the prompt word to be detected may be abnormal, and further, the adversarial perturbation method is used to process the prompt word to be detected to obtain a perturbed prompt word.

[0080] First, the prompt word to be detected is standardized to obtain a standardized prompt word to be detected. Taking the input string prompt as an example, the standardized processing of the input string prompt includes the following steps:

[0081] 1) converting the input string into lowercase letters;

[0082] 2) deleting redundant spaces (including spaces at the beginning and end of the string, replacing multiple consecutive spaces with a single space, and deleting spaces between special characters);

[0083] 3) only keeping the first of the continuous multiple special characters;

[0084] Then, the standardized to-be-detected prompt is randomly processed to obtain a disturbed prompt, and the random processing method includes but is not limited to a BPE-Dropout method. In this embodiment, the BPE-Dropout method is taken as an example, the BPE-D value is selected as 0.4, p% of the BPE is randomly discarded and combined to generate a random word segmentation, and a disturbed prompt corresponding to the input string prompt is obtained.

[0085] In an optional implementation, if the perplexity does not exceed the second threshold, the similarity between the to-be-detected prompt and each reference prompt in the vector database is determined; and if the maximum similarity in the obtained multiple similarities exceeds a third threshold, the to-be-detected prompt is abnormal.

[0086] In an optional implementation, the reference prompts in the vector database are machine-generated first-type prompts and / or artificially designed second-type prompts.

[0087] Specifically, the artificially designed prompt injection attack is generally semantically smooth, and it is difficult to directly detect the prompt abnormality through the perplexity, so the to-be-detected prompt is further detected through a similarity retrieval method. The similarity retrieval method includes a vector database comparison method, an Approximate Nearest Neighbor (ANN), a Locality-Sensitive Hashing (LSH), a Product Quantization (PQ), and a Hierarchical Navigable Small World (HNSW). In this embodiment, the vector database comparison method is taken as an example for illustration. The mainstream vector databases include pinecone, milvus, annoy, and the like.

[0088] The construction process of the vector database is specifically as follows:

[0089] Step 1: Prepare an initial corpus.

[0090] Specifically, the initial corpus includes two types: harmful behaviors and instructions, i.e., machine-generated first-type prompts; and artificially designed second-type prompts, wherein the harmful behaviors and instructions include but are not limited to various problems and instructions in the data set AdvBench and Safety-Prompts, and the artificially designed second-type prompts refer to artificially designed prompts for bypassing the safety protection measures of a large language model, i.e., the safety restrictions of the large language model are removed by designing the prompts, so as to guide or control the large language model to generate harmful content that is not in compliance or should be shielded.

[0091] Step 2: Text vectorization.

[0092] Specifically, each reference prompt in the initial corpus is converted into a fixed-length vector using a text embedding model.

[0093] If the confusion degree does not exceed the second threshold, the similarity between the to-be-detected prompt and each reference prompt in the vector database is determined, wherein the vector database has two, both of which have a corresponding similarity threshold, the reference prompts in the first vector database are machine-generated first-type prompts, and the reference prompts in the second vector database are artificially designed second-type prompts. The vector database is selected according to the demand, the similarity between the to-be-detected prompt and each reference prompt in the vector database is determined, for example, scheme one: the first vector database is selected, and the similarity between the to-be-detected prompt and each reference prompt in the first vector database is determined. Scheme two: the second vector database is selected, and the similarity between the to-be-detected prompt and each reference prompt in the second vector database is determined. Scheme three: the first vector database and the second vector database are selected, and the similarity between the to-be-detected prompt and each reference prompt in the first vector database and the second vector database is determined. The maximum similarity in the plurality of similarities is obtained, and if the maximum similarity exceeds a third threshold, the to-be-detected prompt is abnormal.

[0094] As another implementation, the number of times that the plurality of similarities exceed the third threshold is counted, and if the number of times that the plurality of similarities exceed the third threshold meets a preset standard, the to-be-detected prompt is abnormal.

[0095] In the above implementation, by determining the similarity between the to-be-detected prompt and each reference prompt in the vector database, if the maximum similarity in the plurality of similarities exceeds the third threshold, the to-be-detected prompt is abnormal, and the reference prompts in the vector database are machine-generated first-type prompts and / or artificially designed second-type prompts. Because the vector database has a special vector retrieval query function, the existing or possible abnormal prompts are embedded and stored in the vector database, which not only can effectively prevent similar attacks, but also can effectively detect prompts that can bypass large language model security measures.

[0096] In an optional implementation, the to-be-detected prompt is converted into a to-be-detected prompt vector by a text embedding model; and the similarity between the to-be-detected prompt and each reference prompt is determined based on the distance between the to-be-detected prompt vector and the prompt vector of each reference prompt.

[0097] Specifically, the text embedding model includes: Word2Vec (word to vector model), BERT (Bidirectional Encoder Representations from Transformers), and FastText (shallow network model). The similarity measurement method includes: Euclidean distance, Hamming distance, cosine similarity distance, and Manhattan distance. Embodiments of the present application are described by taking the cosine similarity distance as an example.

[0098] The text embedding model is used to convert the to-be-detected prompt word into a to-be-detected prompt word vector, calculate the cosine similarity distance between the to-be-detected prompt word vector and the prompt word vector of each reference prompt word, determine the similarity between the to-be-detected prompt word and each reference prompt word according to the cosine similarity distance, and the calculation formula of the cosine similarity distance is shown in formula (2):

[0099] Wherein, A and B are vectors of two reference prompt words, and θ is the included angle of the two reference prompt words.

[0100] In step 203, the response semantics of the to-be-detected prompt word and the perturbed response semantics corresponding to the perturbed prompt word are obtained based on the prediction of the large language model.

[0101] Specifically, for a question that can be answered, the large language model usually starts with an affirmative sentence such as "Sure, here's" or "Of course!", and for a question that cannot be answered, it usually starts with a negative sentence such as "I'm sorry" or "As an AI assistant". Therefore, whether the perturbation is effective, i.e., whether the prompt is abnormal, can be determined according to the comparison of the responses of the large language model to the prompt before and after the perturbation.

[0102] The to-be-detected prompt word and the perturbed prompt word are input into the large language model to obtain the response semantics of the to-be-detected prompt word and the perturbed response semantics corresponding to the perturbed prompt word. For example, the to-be-detected prompt word prompt and the perturbed prompt word prompt_disturbed are input into the large language model to obtain the response semantics response and the perturbed response semantics response_disturbed.

[0103] In an optional embodiment, the temperature coefficient of the large language model is not more than 0.6; the temperature coefficient is used to measure the stability of the response output by the large language model.

[0104] Specifically, the large language model is a prediction of the next word, and its output is a probability distribution p i In the last softmax layer, the probability distribution p iSee calculation formula (3):

[0105] Among them, the temperature coefficient T can be x i (logits) adjusted to x i Because the temperature coefficient can affect the diversity of the large language model's output, the lower the temperature coefficient value, the more stable the model output, and the higher the temperature coefficient value, the greater the diversity of the model output. In order to reduce the randomness of the large language model's output and effectively determine the impact of disturbance processing on the prompt word to be detected, the temperature coefficient of the large language model in the embodiment of the present application is set to no more than 0.6.

[0106] In the above embodiment, by setting the temperature coefficient of the large language model to a smaller value (not exceeding 0.6), the output of the large language model can be made relatively stable, the randomness of the output response can be reduced, and the accuracy of the judgment of the influence of the disturbance mode on the prompt word can be improved.

[0107] Step 204: If the response semantics are inconsistent with the disturbance response semantics, the prompt word to be detected is abnormal.

[0108] In an embodiment of the present application, by obtaining the prompt word to be detected and the perplexity of the prompt word to be detected, it is determined that the perplexity exceeds the second threshold but does not exceed the first threshold, and then the prompt word to be detected is processed in an adversarial perturbation manner to obtain the perturbation prompt word, and then the response semantics of the prompt word to be detected and the perturbation response semantics corresponding to the perturbation prompt word are predicted based on the large language model. If the response semantics and the perturbation response semantics are inconsistent, the prompt word to be detected is abnormal. Since the perplexity detection sets a double threshold, and after the prompt word to be detected is perturbated, the response semantics and the perturbation response semantics can be compared to determine whether the prompt word is abnormal, thereby effectively improving the accuracy of abnormality detection of the prompt word and enhancing the security of the large language model.

[0109] In an optional implementation, a vector database for affirmative sentences and a vector database for negative sentences are constructed, and semantic analysis is performed on the response semantics and the disturbance response semantics based on the vector databases. If the response semantics and the disturbance response semantics are affirmative sentence responses and / or negative sentence responses, respectively, the response semantics and the disturbance response semantics are inconsistent, and the prompt word to be detected is abnormal.

[0110] Under the above implementation mode, by processing the prompt word to be detected in an adversarial perturbation manner and performing semantic comparative analysis on the response semantics and perturbation response semantics before and after the prompt word to be detected is disturbed, it is possible to re-determine whether the prompt word to be detected has an abnormality, which is conducive to improving the accuracy of abnormality detection of the prompt word.

[0111] In an optional implementation, after determining that the prompt word to be detected is abnormal, the prompt word to be detected is saved in the vector database as a second type of prompt word.

[0112] Specifically, the prompt word to be detected includes the following abnormal cases: the prompt word to be detected contains adversarial text, the prompt word to be detected contains malicious instructions, and the prompt word to be detected contains artificially designed second-type prompt words.

[0113] In the above embodiment, after determining the abnormality of the prompt word to be detected, the prompt word to be detected is saved as a second-type prompt word in the vector database, which can effectively prevent similar prompt word attacks from occurring again in the future.

[0114] In order to better explain the embodiments of the present application, referring to FIG. 3, the embodiments of the present application provide a flowchart of a prompt word abnormality detection method, which specifically includes the following contents:

[0115] The perplexity of the prompt word to be detected is obtained by inputting the prompt word to be detected into the perplexity detection module. If the perplexity exceeds a first threshold value, the prompt word to be detected is abnormal, and the prompt word to be detected is added to the vector database. If the perplexity does not exceed the first threshold value and exceeds a second threshold value, the prompt word to be detected is disturbed in an adversarial disturbance manner to obtain a disturbed prompt word, and the prompt word to be detected and the disturbed prompt word are respectively input into a large language model to obtain a response semantic of the prompt word to be detected and a disturbed response semantic corresponding to the disturbed prompt word.

[0116] The response semantic and the disturbed response semantic are analyzed and compared. If the response semantic and the disturbed response semantic are inconsistent, the prompt word to be detected is abnormal, and the prompt word to be detected is added to the vector database. If the response semantic and the disturbed response semantic are consistent, the prompt word to be detected is normal.

[0117] If the perplexity does not exceed the second threshold value, the similarity between the prompt word to be detected and each reference prompt word in the vector database is determined. If the maximum similarity in the obtained multiple similarities exceeds a third threshold value, the prompt word to be detected is abnormal, and the prompt word to be detected is added to the vector database. If the maximum similarity in the multiple similarities does not exceed the third threshold value, the prompt word to be detected is normal.

[0118] In addition, the embodiments of the present application are explained by taking a text adversarial attack method as an example.

[0119] For example, the GCG method input prompt word to be detected is obtained by perplexity detection, and the perplexity exceeds threshold 1, and the prompt word to be detected is directly considered to be abnormal; the AutoDAN method input prompt word to be detected is obtained by perplexity detection, and the perplexity does not exceed threshold 1 but exceeds threshold 2, enters the prompt word disturbance processing process, in the input standardization processing, part of the special characters (such as “\”) are deleted, and then the standardized prompt word to be detected is randomly disturbed to obtain a disturbed prompt word, and the large language model respectively gives a positive semantic response and a negative semantic response to the prompt word to be detected and the disturbed prompt word, and then the AutoDAN method input prompt word to be detected is abnormal, and is saved in the vector database.

[0120] Based on the same technical concept, referring to FIG. 4, the embodiment of the present application provides an abnormality detection device for a prompt word, comprising:

[0121] The acquisition module 401 is configured to acquire a prompt word to be detected and a perplexity of the prompt word to be detected.

[0122] The prompt word disturbance module 402 is configured to, if the perplexity exceeds a second threshold and does not exceed a first threshold, process the prompt word to be detected by using an adversarial disturbance manner to obtain a disturbed prompt word.

[0123] The response comparison module 403 is configured to obtain a response semantic of the prompt word to be detected and a disturbed response semantic corresponding to the disturbed prompt word based on a large language model prediction.

[0124] The abnormality detection module 405 is configured to, if the response semantic and the disturbed response semantic are inconsistent, the prompt word to be detected is abnormal.

[0125] In an optional implementation, the device further comprises a similarity comparison module 404.

[0126] The similarity comparison module 404 is specifically configured to:

[0127] If the perplexity does not exceed the second threshold, determine a similarity between the prompt word to be detected and each reference prompt word in the vector database.

[0128] If a maximum similarity in the obtained multiple similarities exceeds a third threshold, the prompt word to be detected is abnormal.

[0129] In an optional implementation, the similarity comparison module 404 is further configured to:

[0130] Convert the prompt word to be detected into a prompt word to be detected vector by using a text embedding model.

[0131] Determine the similarity between the prompt word to be detected and each reference prompt word based on a distance between the prompt word to be detected vector and a prompt word vector of each reference prompt word.

[0132] In an optional implementation, the method further includes a prompt word storage module 406.

[0133] The prompt word storage module 406 is specifically configured to:

[0134] save the to-be-detected prompt word as a second type of prompt word in a vector database.

[0135] In the embodiments of the present application, by obtaining the to-be-detected prompt word and the perplexity of the to-be-detected prompt word, it is determined that the perplexity exceeds the second threshold and does not exceed the first threshold, and then the to-be-detected prompt word is processed in an adversarial perturbation manner to obtain a perturbed prompt word, and then the response semantics of the to-be-detected prompt word and the perturbed response semantics corresponding to the perturbed prompt word are obtained based on the large language model prediction. If the response semantics and the perturbed response semantics are inconsistent, the to-be-detected prompt word is abnormal. Since the perplexity detection sets double thresholds, and after the perturbation processing of the to-be-detected prompt word, the comparison of the response semantics and the perturbed response semantics can determine whether the prompt word is abnormal again, thereby effectively improving the accuracy of the abnormal detection of the prompt word and enhancing the security of the large language model.

[0136] Based on the same technical concept, the embodiments of the present application provide a computer device. The computer device can be the terminal device and / or the detection system shown in FIG. 1. As shown in FIG. 5, the computer device includes at least one processor 501 and a memory 502 connected with the at least one processor. In the embodiments of the present application, the specific connection medium between the processor 501 and the memory 502 is not limited, and in FIG. 5, the processor 501 and the memory 502 are connected through a bus as an example. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0137] In the embodiments of the present application, the memory 502 stores instructions executable by the at least one processor 501. The at least one processor 501 can execute the steps of the above-mentioned prompt word abnormal detection method by executing the instructions stored in the memory 502.

[0138] The processor 501 is the control center of the computer device, can connect various parts of the computer device by using various interfaces and lines, and can realize the prompt word anomaly detection by running or executing instructions stored in the memory 502 and calling data stored in the memory 502. Optionally, the processor 501 can include one or more processing units, and the processor 501 can integrate an application processor and a modem processor. The application processor mainly processes an operating system, a user interface, and an application program, and the modem processor mainly processes wireless communication. It can be understood that the modem processor can also not be integrated into the processor 501. In some embodiments, the processor 501 and the memory 502 can be implemented on the same chip, and in some embodiments, they can also be respectively implemented on independent chips.

[0139] The processor 501 can be a general-purpose processor, for example, a central processing unit (CPU), a digital signal processor, an application specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as hardware processor execution or executed by a combination of hardware and software modules in the processor.

[0140] The memory 502, as a non-volatile computer readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 502 can include at least one type of storage medium, for example, can include flash memory, hard disk, multimedia card, card type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. The memory 502 is any other medium capable of carrying or storing desired program code in the form of instructions or data structures and capable of being accessed by a computer device, but is not limited thereto. The memory 502 in the embodiments of the present application can also be a circuit or any other device capable of realizing a storage function, used to store program instructions and / or data.

[0141] Based on the same inventive concept, the embodiments of the present application provide a computer readable storage medium storing a computer program executable by a computer device, which, when executed on the computer device, causes the computer device to perform the steps of the above prompt word anomaly detection method.

[0142] Based on the same inventive concept, the embodiments of the present application provide a computer program product, which comprises a computer program stored on a computer readable storage medium, the computer program comprising program instructions, which, when executed by a computer device, cause the computer device to perform the steps of the above prompt word anomaly detection method.

[0143] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0144] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks.

[0145] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks.

[0146] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks.

[0147] Although preferred embodiments of the application have been described herein, it will be apparent to those skilled in the art that various modifications can be made within the scope of the application. Accordingly, it is intended that all content of the above description and all variations and modifications be included within the scope of the application. The above description is intended to be illustrative and not restrictive. Many other potential applications of the application will be apparent to those skilled in the art.

[0148] Obviously, many modifications and variations of the present application are possible in light of the above teachings. It is, therefore, to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A method for detecting abnormalities in prompt words, characterized in that: include: Obtaining a prompt word to be detected and a perplexity of the prompt word to be detected; If the perplexity exceeds the second threshold and does not exceed the first threshold, the prompt word to be detected is processed in an adversarial perturbation manner to obtain a perturbation prompt word; Obtaining the response semantics of the prompt word to be detected and the disturbance response semantics corresponding to the disturbance prompt word based on the large language model prediction; If the response semantics are inconsistent with the disturbance response semantics, the prompt word to be detected is abnormal.

2. The method according to claim 1, wherein After obtaining the prompt word to be detected and the perplexity of the prompt word to be detected, the method further includes: If the perplexity does not exceed a second threshold, determining the similarity between the prompt word to be detected and each reference prompt word in the vector database; If the maximum similarity among the obtained multiple similarities exceeds the third threshold, the prompt word to be detected is abnormal.

3. The method according to claim 2, wherein The reference prompt words in the vector database are: machine-generated first-category prompt words, and / or manually designed second-category prompt words.

4. The method according to claim 2, wherein Determining the similarity between the prompt word to be detected and each reference prompt word in the vector database includes: Converting the prompt word to be detected into a prompt word vector to be detected through a text embedding model; Based on the distance between the prompt word vector to be detected and the prompt word vector of each reference prompt word, the similarity between the prompt word to be detected and each reference prompt word is determined.

5. The method according to claim 1, wherein The temperature coefficient of the large language model does not exceed 0.6; the temperature coefficient is used to measure the stability of the output response of the large language model.

6. The method according to claim 3, wherein After determining that the prompt word to be detected is abnormal, the following operations are also included: The prompt word to be detected is stored in the vector database as a second type of prompt word.

7. The method according to any one of claims 1 to 6, characterized in that: The adversarial perturbation methods include: synonym replacement, word order disruption, subword rearrangement, and morphological change.

8. A device for detecting abnormalities in prompt words, characterized in that: include: An acquisition module, configured to acquire a prompt word to be detected and a perplexity of the prompt word to be detected; a prompt word perturbation module, configured to process the prompt word to be detected using an adversarial perturbation method to obtain a perturbed prompt word if the perplexity exceeds the second threshold but does not exceed the first threshold; a response comparison module, configured to obtain, based on a large language model, a response semantic of the prompt word to be detected and a disturbance response semantic corresponding to the disturbance prompt word; The anomaly detection module is used to determine that the prompt word to be detected is abnormal if the response semantics are inconsistent with the disturbance response semantics.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that It stores a computer program that can be executed by a computer device. When the program is run on the computer device, the computer device executes the steps of any one of the methods according to claims 1 to 7.

11. A computer program product, characterized in that The computer program product comprises a computer program stored on a computer-readable storage medium, wherein the computer program comprises program instructions. When the program instructions are executed by a computer device, the computer device is caused to perform the steps of the method according to any one of claims 1 to 7.

Images