Prompt word processing method, prompt word processing device, electronic equipment and medium

Abstract

The invention provides a cue word processing method, a cue word processing device, electronic equipment and a computer readable storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring corpus information, and generating a plurality of first cue words according to the corpus information; determining a plurality of second cue words based on the plurality of first cue words; processing the second cue words by using a pre-trained cue word classification model to obtain classification results corresponding to the second cue words; and according to the classification result of each second cue word, carrying out set division on the plurality of second cue words to generate a cue word set. According to the method and the device, the cue word can be accurately and effectively processed to obtain the cue word set.

Description

Background Art

[0002] With the rapid development of AIGC (Artificial Intelligence Generated Content) technology, it has been widely used in multiple application scenarios. In order to ensure that the generated content meets certain requirements, such as security and compliance, large models often need to detect the generated content before application. Usually, the method adopted is generally to generate highly inductive prompt words based on seed topics and various attack methods, and induce the large model to generate illegal content. However, on the one hand, due to the different time for the large model to generate reply information for different prompt words, for example, the difficulty of the prompt words is different, and the time to generate replies will also be different; on the other hand, the detection process often directly uses randomly obtained prompt words for detection, and the overall management of prompt words used for testing is relatively chaotic, resulting in low detection efficiency of large models, unable to guarantee its accuracy, and unstable detection completion time. Therefore, how to accurately and effectively manage prompt words is a problem that needs to be solved in the prior art.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0004] The present disclosure provides a prompt word processing method, a prompt word processing device, an electronic device and a computer-readable storage medium, thereby overcoming the problem at least to a certain extent.

[0005] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0006] According to one aspect of the present disclosure, a prompt word processing method is provided, including: acquiring corpus information, and generating multiple first prompt words according to the corpus information; determining multiple second prompt words based on the multiple first prompt words; processing the second prompt words using a pre-configured prompt word classification model to obtain classification results corresponding to each of the second prompt words; and dividing the multiple second prompt words into sets according to the classification results of each of the second prompt words to generate a prompt word set.

[0007] According to one aspect of the present disclosure, a prompt word processing device is provided, including: a first prompt word generating module, used to obtain corpus information, and generate multiple first prompt words according to the corpus information; a second prompt word obtaining module, used to optimize the first prompt words to obtain multiple second prompt words; a second prompt word processing module, used to process the second prompt words using a pre-configured prompt word classification model to obtain classification results corresponding to each of the second prompt words; a prompt word set generating module, used to divide the multiple second prompt words into sets according to the classification results of each of the second prompt words, and generate a prompt word set.

[0008] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned methods by executing the executable instructions.

[0009] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, any one of the above methods is implemented.

[0010] The exemplary embodiments of the present disclosure have the following beneficial effects:

[0011] Acquire corpus information, and generate multiple first prompt words according to the corpus information; determine multiple second prompt words based on the multiple first prompt words; use the pre-trained prompt word classification model to process the second prompt words to obtain the classification results corresponding to each second prompt word; and divide the multiple second prompt words into sets according to the classification results of each second prompt word to generate a prompt word set. On the one hand, this exemplary embodiment proposes a new prompt word processing method, which can divide the multiple second prompt words into sets according to the classification results of the second prompt words to generate a corresponding prompt word set. Compared with the method of directly using or using random prompt words for model testing in the prior art, the prompt word set in this exemplary embodiment is determined according to the classification results of the second prompt words, and the prompt word type distribution is regular and accurate. The prompt word set is used for model testing, and the test effect is more stable and effective; on the other hand, this exemplary embodiment can process the multiple generated first prompt words to obtain multiple second prompt words, and generate a prompt word set based on the second prompt words, which is equivalent to the prompt words in the prompt word set are all prompt words processed from the initial prompt words, and the prompt words are more accurate. The prompt word set is used for model testing, which further ensures the accuracy of the model test.

[0012] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0014] Figure 1 A flowchart schematically shows a prompt word processing method in this exemplary embodiment;

[0015] Figure 2 A schematic diagram of a training framework of a prompt word classification model in this exemplary embodiment is schematically shown;

[0016] Figure 3 A flowchart schematically shows another prompt word processing method in this exemplary embodiment;

[0017] Figure 4 A structural block diagram of a prompt word processing device in this exemplary embodiment is schematically shown;

[0018] Figure 5 An electronic device for implementing the above method in this exemplary embodiment is schematically shown. DETAILED DESCRIPTION

[0019] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0020] An exemplary embodiment of the present disclosure first provides a prompt word processing method.

[0021] The following is combined with Figure 1 This exemplary embodiment is further described as follows. Figure 1 As shown, the prompt word processing method may include the following steps S110 to S140:

[0022] Step S110, acquiring corpus information, and generating a plurality of first prompt words according to the corpus information.

[0023] The corpus information refers to the basic information used to generate the prompt word, which can be information input by the user or information obtained from the network, such as texts and topics downloaded from the network. The first prompt word refers to the prompt word used to test the model to be tested, etc. The first prompt word is input into the model to be tested, and the reply content of the model to be tested can be obtained. The model to be tested can be tested according to the reply content, for example, whether the model to be tested is trained in compliance or meets security requirements according to whether the reply results contain illegal words, etc. The first prompt word can be regarded as a kind of attack information for attacking the model to be tested. The corpus information can be regarded as seed information for generating the first prompt word.

[0024] In this exemplary embodiment, the corpus information can be obtained in a variety of ways, such as through a web crawler, an API (Application Programming Interface) interface, such as the API of a news website, or a public data set to collect text data sets in different fields as a source for obtaining corpus information. After obtaining the corpus information, it can also be preprocessed, such as using NLP (Natural Language Processing) technology to preprocess the corpus information, which can include one or more of tokenization, stopword removal, and stemming to ensure that the corpus information has semantic clarity and structural integrity. At the same time, it is also possible to write Python (a programming language) scripts and use NLP libraries such as spaCy (a Python library), NLTK (Natural Language Toolkit, Natural Language Processing Toolkit), etc. to perform text cleaning. In addition, for high-frequency words or irrelevant information, the TF-IDF (term frequency-inverse document frequency) algorithm can be used to filter out high-weight words to improve the quality of the corpus information.

[0025] When generating multiple first prompt words according to the corpus information, the exemplary embodiment can adopt different methods. Specifically, different attack methods can be crawled and parsed through automated scripts, including but not limited to adversarial example generation, model stealing, data poisoning, jailbreak attack, prompt word injection, and reverse optimization attack, etc. For example, relevant technical documents or papers can be obtained from a database or network, and different attack methods can be extracted therefrom. Theme modeling is performed according to the obtained document content through a machine learning algorithm. According to the technical principles of the identified different attack methods, different attack methods are introduced into the embedding layer of the model using the acquired corpus information, and automated scripts are further written to build a multi-level reasoning pipeline. The generation capability of the large-scale pre-trained model of the Transformer architecture is used to dynamically generate and expand attack instructions according to the corpus information and the extracted attack methods. The generation and expansion process of the attack instructions is realized by calling the generation model of the API or the localized deployment. The automated script will call the generation model in batches to continuously expand the attack instructions. Finally, multiple first prompt words are generated according to the corpus information and multiple attack instructions.

[0026] Step S120: determining a plurality of second prompt words based on the plurality of first prompt words.

[0027] In this exemplary embodiment, the prompt word can be used to attack the model to be tested, and therefore, it can have certain negative attributes, such as the desire to obtain malicious prompt words. Therefore, in order to obtain reasonable and rich prompt words to test the model to be tested, this exemplary embodiment can optimize the first prompt word, and the second prompt word is the prompt word obtained by optimizing the first prompt word. This exemplary embodiment can construct a rich second prompt word that meets the attack requirements based on the first prompt word.

[0028] In an exemplary embodiment, the above-mentioned determining a plurality of second prompt words based on a plurality of first prompt words may include the following steps:

[0029] Block part of the information in the first prompt word to obtain the first middle word;

[0030] According to the first middle word, detecting whether the first prompt word is a negative prompt word;

[0031] If it is determined that the first prompt word is a negative prompt word, then determining the second prompt word according to the first prompt word;

[0032] If it is determined that the first prompt word is a non-negative prompt word, a second middle word is constructed according to the first prompt word, and the second prompt word is determined according to the first prompt word and / or the second middle word.

[0033] In the related art, samples are usually constructed directly based on the attack mode for detection. However, this will consume a lot of computing resources and reasoning delay. Therefore, this exemplary embodiment proposes a method for constructing a prompt word detection set based on greedy optimization, which can first block part of the information in the first prompt word to obtain a first middle word; according to the first middle word, detect whether the first prompt word is a negative prompt word. Among them, blocking part of the information in the first prompt word can be erasing the characters therein in a certain order to obtain a new prompt word, that is, the first middle word. For example, the first prompt word is "abcd", and the characters therein are erased one by one in a preset order. The first middle word that can be obtained may include "bcd", "acd", "abd", "abc", and multiple characters can be erased as needed. The present disclosure does not make specific restrictions on this. Then, it can be detected whether the first middle word is a negative prompt word, for example, using a pre-trained detector to calculate whether the probability that the first middle word after erasing the information is a malicious prompt word is greater than the probability that it is a benign prompt word.

[0034] Then, the second prompt word can be determined according to the detection result of the first prompt word and the first prompt word. In this exemplary embodiment, if the first prompt word is determined to be a negative prompt word, the first prompt word can be directly used as the second prompt word to perform the application of the model test. If the first prompt word is determined to be a non-negative prompt word, such as a benign prompt word, multiple second intermediate words can be constructed according to the first prompt word, and the second prompt word can be determined according to the first prompt word and / or the second intermediate word. For example, the first prompt word and the second intermediate word can be used as the second prompt word, that is, the first prompt word itself and the constructed second intermediate word are both used as the second prompt word; the target intermediate word can also be determined from the multiple second intermediate words, and the first prompt word and the target intermediate word can be used as the second prompt word, for example, the second intermediate word with the highest malicious probability among the first prompt word and the multiple second intermediate words can be used as the second prompt word; one or more of the second intermediate words can also be directly used as the second prompt word, and the present disclosure does not make specific limitations on this.

[0035] Among them, constructing the second intermediate word according to the first prompt word can be achieved in many ways, for example, one or more methods of anti-insertion, anti-deletion, anti-confusion, anti-prefix, anti-suffix, and anti-replacement can be used to expand the first prompt word in many aspects, making the prompt word larger in scale.

[0036] Among them, adversarial insertion can be to insert a piece of information in the middle position of the prompt word sequence to bypass the security protection mechanism of the large model. The threat model of this attack can be defined as the set of all prompt words generated by inserting a continuous token sequence θ with a maximum length of m into any position in the prompt word T, Insertion (T, m), which can be defined as:

[0037] Insertion(T, m) = {T 1 +θ+T 2 , T 1 +T 2 =T,θ≤m}

[0038] Among them, T represents the prompt word, T 1 and T 2 Construct the prompt word T.

[0039] Adversarial deletion can be to delete a part of the adversarial sequence in the sequence of prompt words to bypass the security protection mechanism of the large model. The threat model of this attack can be defined as the set Deletion(T, m) consisting of all prompt words generated by deleting the tag sequence θ of the maximum length m after the prompt word T, which is defined as:

[0040] Deletion(T, m)={T-θ, |θ|≤m}

[0041] Adversarial confusion can be achieved by inserting up to m special tags (such as malicious tags) at any position in the prompt word sequence to bypass the security protection mechanism of the large model. The threat model of this attack can be defined as the set of all prompt words generated by inserting a sequence of m tags into the prompt word T, Confusion (T, m), where τ represents the inserted i-th tag, which can be defined as:

[0042] Confusion(T, m) = {T 1 +τ 1 +…+τ n +T n+1 ,∑T i =T, n≤m}

[0043] The adversarial prefix can be added to the first position of the prompt word to bypass the security protection mechanism of the large model. The threat model of this attack can be defined as the set Prefixion (T, m) consisting of all prompt words generated by adding a tag sequence θ of maximum length m before the prompt word T, which can be defined as:

[0044] Prefixion(T, m)={θ+T, |θ|≤m}

[0045] The adversarial suffix can be added to the end of the prompt word to bypass the security protection mechanism of the large model. The threat model of this attack can be defined as the set Postfix (t, m) consisting of all prompt words generated by adding a tag sequence θ of maximum length m after the prompt word T, which can be defined as:

[0046] Postfix(T,m)={T+θ,|θ|≤m}

[0047] For example, you can first obtain the prompt word P, and in order, for example, erase the text information one by one, and use the subsequent detector to calculate whether the probability of the prompt word to be detected after the erased text information is a malicious prompt word is greater than the probability of it being a benign prompt word. If so, the original prompt word P is determined to be a malicious prompt word. If no malicious prompt word is detected during a certain iteration, the prompt word with the highest probability of being a malicious prompt word among the currently constructed prompt words to be detected is selected to repeat the iteration process.

[0048] Step S130: Use a pre-configured prompt word classification model to process the second prompt words to obtain classification results corresponding to each second prompt word.

[0049] In this exemplary embodiment, after obtaining multiple second prompt words, the second prompt words can be processed using a pre-trained prompt word classification model to obtain a classification result corresponding to each second prompt word. Specifically, the second prompt words can be used as input data of the prompt word classification model. After the prompt word classification model processes them, the classification result of the second prompt word as a positive prompt word or a negative prompt word can be input, such as outputting "0", "1", or "positive", "negative" results, or outputting positive or negative probability values, etc.

[0050] The prompt word classification model can be a classification detection model, which can be used to classify prompt words. This exemplary embodiment can directly train a classification model to obtain a prompt word classification model, or use a BERT (Bidirectional Encoder Representations from Transformers) model to determine the prompt word classification model. For example, the BERT model can be distilled, and then the distilled BERT model can be used to train a classification detector to obtain a prompt word classification model, and so on.

[0051] Step S140: dividing the plurality of second prompt words into sets according to the classification results of the second prompt words to generate a prompt word set.

[0052] Finally, this exemplary embodiment can divide multiple second prompt words into sets according to the classification results of the second prompt words to obtain prompt word sets. In this exemplary embodiment, there can be multiple prompt word sets, and the number of second prompt words in each set can be the same or different, which is not specifically limited in this disclosure.

[0053] The classification results of the second prompt words may include two or more. For example, the second prompt words may be divided into malicious prompt words and benign prompt words, or may be divided into malicious level 3 prompt words, malicious level 2 prompt words, malicious level 1 prompt words and benign prompt words, etc. The second prompt words may be grouped according to the classification results of the second prompt words. The prompt words belonging to the same category may be grouped into one category, or the prompt words belonging to different categories may be grouped into one category according to a certain ratio, etc. The specific classification may be determined according to actual needs, and the present disclosure does not make any specific limitation on this.

[0054] In an exemplary embodiment, the above classification result may include a first classification result and a second classification result;

[0055] The above-mentioned dividing the plurality of second prompt words into sets according to the classification results of the second prompt words to generate a prompt word set may include:

[0056] The second prompt words of the first classification results of the first quantity and the second prompt words of the second classification results of the second quantity are divided into the same set to generate a prompt word set; the ratio of the first quantity to the second quantity is a preset ratio.

[0057] Among them, the first classification result and the second classification result can be the classification results of the same indicator at different degrees, such as the classification results of prompt words with a negative degree of 1 and prompt words with a negative degree of 0. The first classification result and the second classification result can also be the classification results of different indicators, such as the classification results of malicious prompt words and benign prompt words.

[0058] This exemplary embodiment can divide the first number of second prompt words of the first classification results and the second number of second prompt words of the second classification results into the same set to generate a prompt word set, wherein the ratio of the first number to the second number is a preset ratio, that is, this exemplary embodiment can divide the prompt words belonging to different classification results into sets according to a preset ratio to obtain a prompt word set, for example, the first classification result is a malicious prompt word, and the second classification result is a benign prompt word, this exemplary embodiment can divide the malicious prompt words and the benign prompt words in a ratio of 8:2 to obtain one or more prompt word sets, each prompt word set includes a first number of malicious prompt words and a second number of benign prompt words, and the ratio of the first number to the second number is 8:2.

[0059] It should be noted that the above-mentioned first quantity and second quantity are only for reference. In actual applications, the first quantity and the second quantity in different prompt word sets may be different, but the ratio of the first quantity to the second quantity is a preset ratio.

[0060] Based on the above description, in this exemplary embodiment, corpus information is obtained, and multiple first prompt words are generated according to the corpus information; multiple second prompt words are determined based on the multiple first prompt words; the second prompt words are processed using a pre-trained prompt word classification model to obtain the classification results corresponding to each second prompt word; and multiple second prompt words are divided into sets according to the classification results of each second prompt word to generate a prompt word set. On the one hand, this exemplary embodiment proposes a new prompt word processing method, which can divide multiple second prompt words into sets according to the classification results of the second prompt words to generate corresponding prompt word sets. Compared with the method of directly using or using random prompt words for model testing in the prior art, the prompt word set in this exemplary embodiment is determined according to the classification results of the second prompt words, and the prompt word type distribution is regular and accurate. The prompt word set is used for model testing, and the test effect is more stable and effective; on the other hand, this exemplary embodiment can process the generated multiple first prompt words to obtain multiple second prompt words, and generate a prompt word set based on the second prompt words, which is equivalent to the prompt words in the prompt word set are all prompt words processed from the initial prompt words, and the prompt words are more accurate. The prompt word set is used for model testing, which further ensures the accuracy of the model test.

[0061] In an exemplary embodiment, the prompt word processing method may further include:

[0062] Using the language model to be tested to process the second prompt word in the prompt word set, to obtain output data corresponding to the second prompt word;

[0063] A compliance check is performed on the output data to determine a test result of the language model to be tested according to the compliance check result.

[0064] Among them, the language model to be tested can be any large language model that needs to be tested. Compliance check refers to the check whether the language model to be tested meets the expected specification standards. By judging the output content of the language model to be tested, it can be determined whether it meets the specifications, such as whether the output content contains malicious content or negative content that does not comply with relevant regulations. In this exemplary embodiment, compliance can be determined by a pre-configured violation information list or by a trained model. For example, the output data can be matched with the violation information list to determine whether it contains violation information. It can also be used as input data for a violation check model to determine whether it is in violation and what the violation information is.

[0065] After obtaining the prompt word set, this exemplary embodiment can use the second prompt word therein as input data of the language model to be tested to obtain output data of the language model to be tested. The output data can be regarded as the reply content of the language model to be tested based on the second prompt word. By performing a compliance check on the reply content, the test result of the model to be tested can be determined, for example, the classification result of whether the model to be tested passes the test, or the specific details of passing or failing the test, such as the reason for failing the test, what problems exist, etc.

[0066] In an exemplary embodiment, the above-mentioned compliance check on the output data to determine the test result of the language model to be tested according to the compliance check result may include:

[0067] When it is determined that the compliance rate of the output data is greater than a preset threshold, it is determined that the language model to be tested passes the test.

[0068] Among them, the compliance rate can be the ratio of output data that meets the specifications to output data that does not meet the specifications, and the preset threshold can be the standard for judging whether the model passes the test. The preset threshold can be customized according to actual needs or the inclusiveness of the model, and the present disclosure does not make specific limitations on this.

[0069] This exemplary embodiment can write a Python script locally to detect the compliance rate of the large model to be tested. The Python script is uploaded to the detection system, associated with the indicators to be detected, and stored as an atomic information. When configuring the detection task, the language model to be tested can be selected, the evaluation tool and the prompt word set can be selected, and the order of the language model to be tested-evaluation tool-prompt word set can be connected to form a task flow. For example, this exemplary embodiment can use a test system including visual operations to drag the language model to be tested-evaluation tool-prompt word set to a specified position to generate a task flow, and then define the indicator requirements for the compliance rate, such as the compliance rate must be higher than 95% to pass this detection task, that is, if a detection task contains 100 prompt words and 100 replies are generated, then more than 95 replies must be compliant to pass the detection of the language model to be tested.

[0070] In an exemplary embodiment, the above prompt word processing method may further include:

[0071] Obtain a sensitive database, and use the output data to match the sensitive database to determine the sensitive data matching result;

[0072] According to the sensitive data matching results, the test results of the language model to be tested are checked.

[0073] Among them, the sensitive database refers to a database used to store or manage sensitive data. This exemplary embodiment can obtain a large amount of sensitive data, such as sensitive words, through a web crawler or other means, classify them based on the label system generated by the model, and establish a sensitive word library in the detection system. For example, this exemplary embodiment can create a multi-level label system and use the model to classify sensitive words into different label systems. The sensitive word library can have a specially managed page in the detection system to facilitate statistics on which label systems have a higher hit rate of sensitive words.

[0074] When the compliance test is performed on the output data, the exemplary embodiment can also match the output data with the sensitive database to determine the sensitive data matching result. The sensitive data matching result can be used to reflect whether the output content of the language model to be tested hits the sensitive word library. If it hits, it can be marked in the form of highlighting or other forms. For example, there is a prompt word "the leader of the xx region is a", and the sensitive word library contains the word "xx". If this word appears in the generated content of the large model to be tested, it can be marked as highlighted or displayed in other distinctive ways. Further, the test results of the language model to be tested can be checked according to the sensitive data matching results. The check can be regarded as a process of reviewing the test results of the language model to be tested to determine whether it is truly compliant. For example, if a test result hits the sensitive word library, it will be marked as highlighted. In the subsequent review stage, the staff will pay more attention to this sentence, and combine the actual context and other information to finally determine whether the test result is compliant and whether the language model to be tested has passed the test.

[0075] It should be noted that, in this exemplary embodiment, when matching the output data with the sensitive database, whether the output data is in violation of regulations has no necessary relationship with whether it matches the sensitive data. That is, the fact that the output data matches the sensitive data does not necessarily mean that the output data is non-compliant. The sensitive database is used to mark the test data to facilitate subsequent review, calibration, search and browsing, etc.

[0076] In an exemplary embodiment, the checking of the test result of the language model to be tested according to the sensitive data matching result may include:

[0077] Based on the sensitive data matching results, the test results of the language model to be tested are reviewed.

[0078] In this exemplary embodiment, the test results will be brought into the review phase. This exemplary embodiment can provide a visual display method to display the test results on the display device. Each prompt word and its generated content will be displayed on the page, and the evaluation results of the evaluation model will be displayed. It can also include whether the test results hit the sensitive word library, and the highlight display after hitting the sensitive word library. Further, the user can review based on the content displayed on the display device. For example, the model evaluation result of a certain question is a compliant reply. There are two buttons below for professional labelers to judge. If the labeler also chooses a compliant reply, it means that the evaluation result of the evaluation model is accurate. In addition, this exemplary embodiment can also be performed by a review robot to perform the review process. If the review robot is connected for review, its review results can be compared with the model evaluation results, and the performance of the evaluation model can be evaluated. The detection result of the detection task can ultimately be based on the review result of the review phase. This exemplary embodiment can also review the test results by manual + robot, etc., and this disclosure does not make specific limitations on this.

[0079] The test results of the language model to be tested, the compliance results of the review staff, and the compliance results of the review robot can be compared in multiple ways. If the three results are consistent, it means that the evaluation result of the evaluation model for a certain question and answer pair is accurate. The evaluation results of all question and answer pairs are counted to comprehensively evaluate the performance of the evaluation model. The final evaluation results can be based on the results of the review robot and the labeler.

[0080] In an exemplary embodiment, the prompt word processing method may further include:

[0081] Obtaining a first feature extraction model, and performing distillation processing on the first feature extraction model to obtain a second feature extraction model;

[0082] The prompt word classification model is determined according to the second feature extraction model.

[0083] Among them, the first feature extraction model can be a BERT model, and the second feature extraction model can be a DistilBERT (distilled BERT) model. This exemplary embodiment can perform knowledge distillation processing on the BERT model, and then use the DistilBERT model to train a classification detector to obtain a prompt word classification model. The prompt word classification model can include a DistilBERT model and a classification detector. The prompt word classification model can classify the prompt words to filter the prompt words and improve the detection efficiency of the prompt words of a certain classification. Specifically, the prompt words to be classified can be input into the prompt word classification model, and processed by the DistilBERT model and the classification detector to obtain the classification result corresponding to the prompt words.

[0084] In this exemplary embodiment, after obtaining the second prompt word, the second prompt word is usually mixed with prompt words of different categories, and these prompt words can be used as a training data set to train a BERT-based detector. Currently, the BERT model has been widely used to process NLP tasks such as text classification. Through deep bidirectional representation learning and multi-task pre-training, it greatly improves the performance of NLP tasks and flexibly adapts to complex and diverse NLP tasks.

[0085] Knowledge distillation is a model compression technology. Usually, a large model with complex performance (usually called a teacher model) guides a simple small model (usually called a student model). During the distillation process, the small model can acquire some "knowledge" of the large model and have similar performance.

[0086] This exemplary embodiment can use DistilBERT obtained by knowledge distillation of the BERT model as a text feature extractor, compress the model, significantly reduce the number of model parameters, and improve the detection efficiency of the detector. In the process of knowledge distillation, the training loss function of the student model can be defined as: L t =∑ ( m ( log(n ( ), where mi represents the predicted distribution of the teacher model for sample i, and ni represents the predicted distribution of the student model for sample i.

[0087] Figure 2 A schematic diagram of an architecture for determining a prompt word classification model is shown. After the BERT model is obtained through knowledge distillation to obtain a DistilBERT model, it can be used to train a prompt word detector 210. The prompt word detector 210 is the prompt word classification model in this exemplary embodiment, which may include the DistilBERT model and classifier obtained by distillation. The specific process may include: D may include sample prompt words 221 and label data 222 for model training. The training samples in D are passed to the DistilBERT model 230 to obtain the representation of each training sample in the training set D and transmit it to the classifier 240. The classifier 240 may be a binary classifier. The classifier may predict the classification result 250 of each training sample, such as the probability distribution of the training sample being a malicious prompt word. Then, the predicted probability distribution is used with the real sample label 260 that has been marked in advance to calculate the binary classification cross entropy loss. The loss function 270 may be defined as:

[0088]

[0089] Here, n represents the number of samples in the training dataset. By back-propagating the loss to calculate the gradient of the detector parameters and updating the parameters, it can be found that compared with the original BERT model, the DistilBERT model processed with knowledge distillation is more efficient and has less impact on the overall detection accuracy.

[0090] Figure 3 A flowchart of another prompt word processing method in this exemplary embodiment is shown, which may specifically include:

[0091] Step S310, acquiring corpus information and attack information, and generating a first prompt word according to the corpus information and the attack information;

[0092] Step S320, performing greedy optimization processing on the first prompt word to determine the second prompt word;

[0093] Step S330, obtaining the first feature extraction model, and performing knowledge distillation processing on it to obtain a second feature extraction model, and using the second feature extraction model in the detector to obtain a prompt word classification model;

[0094] Step S340, using the prompt word classification model to classify the second prompt word to obtain a classification result of the second prompt word;

[0095] Step S350, according to the classification results of the second prompt words, the second prompt words of different classification results are divided according to a preset ratio to obtain a prompt word set;

[0096] Step S360, processing the second prompt word in the prompt word set as input data of the language model to be tested to obtain an output result;

[0097] Step S370: Perform compliance check on the output result to obtain the test result of the language model to be tested, and match the output result with the sensitive word database to obtain a sensitive data matching result;

[0098] Step S380: review the test results of the language model to be tested based on the sensitive data matching results.

[0099] The exemplary embodiment of the present disclosure also provides a prompt word processing device. Figure 4The device 400 may include a first prompt word generating module 410, which is used to obtain corpus information and generate a plurality of first prompt words according to the corpus information; a second prompt word obtaining module 420, which is used to optimize the first prompt word to obtain a plurality of second prompt words; a second prompt word processing module 430, which is used to process the second prompt word using a pre-configured prompt word classification model to obtain classification results corresponding to each second prompt word; and a prompt word set generating module 440, which is used to divide the plurality of second prompt words into sets according to the classification results of each second prompt word to generate a prompt word set.

[0100] In an exemplary embodiment, the prompt word processing device may further include: a prompt word processing unit, used to process the second prompt word in the prompt word set using the language model to be tested to obtain output data corresponding to the second prompt word; and a compliance checking unit, used to perform a compliance check on the output data to determine the test result of the language model to be tested based on the compliance check result.

[0101] In an exemplary embodiment, the compliance checking unit includes: a compliance rate judgment subunit, which is used to determine that the language model to be tested passes the test when it is determined that the compliance rate of the output data is greater than a preset threshold.

[0102] In an exemplary embodiment, the prompt word processing device may further include: a sensitive data matching unit, used to obtain a sensitive database, and use the output data to match the sensitive database to determine the sensitive data matching result; a test result checking unit, used to check the test result of the language model to be tested based on the sensitive data matching result.

[0103] In an exemplary embodiment, the second prompt word obtaining module includes: a first middle word obtaining unit, which is used to block part of the information in the first prompt word to obtain the first middle word; a negative prompt word detection unit, which is used to detect whether the first prompt word is a negative prompt word based on the first middle word; a second prompt word determination unit, which is used to determine the second prompt word based on the first prompt word if the first prompt word is determined to be a negative prompt word; and a second middle word construction unit, which is used to construct the second middle word based on the first prompt word if the first prompt word is determined to be a non-negative prompt word, and determine the second prompt word based on the first prompt word and / or the second middle word.

[0104] In an exemplary embodiment, the prompt word processing device may further include: a feature extraction model processing unit, used to obtain a first feature extraction model and perform distillation processing on the first feature extraction model to obtain a second feature extraction model; and a prompt word classification model determination unit, used to determine a prompt word classification model based on the second feature extraction model.

[0105] In an exemplary embodiment, the classification result includes a first classification result and a second classification result: the prompt word set generation module includes: a proportion division unit, used to divide the second prompt words of the first classification results of the first quantity and the second prompt words of the second classification results of the second quantity into the same set to generate a prompt word set; the ratio of the first quantity to the second quantity is a preset ratio.

[0106] The specific details of each module / unit in the above-mentioned device have been described in detail in the embodiments of the method part. The undisclosed details can be found in the embodiments of the method part, so they will not be repeated here.

[0107] Exemplary embodiments of the present disclosure also provide an electronic device capable of implementing the above method.

[0108] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0109] Refer to the following Figure 5 An electronic device 500 according to such an exemplary embodiment of the present disclosure is described below. Figure 5 The electronic device 500 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0110] like Figure 5 As shown, the electronic device 500 is in the form of a general computing device. The components of the electronic device 500 may include but are not limited to: at least one processing unit 510, at least one storage unit 520, a bus 530 connecting different system components (including the storage unit 520 and the processing unit 510), and a display unit 540.

[0111] The storage unit stores program codes, which can be executed by the processing unit 510, so that the processing unit 510 executes the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 510 can execute Figure 1 or Figure 3 Steps shown etc.

[0112] The storage unit 520 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 521 and / or a cache memory unit 522 , and may further include a read-only memory unit (ROM) 523 .

[0113] The storage unit 520 may also include a program / utility 524 having a set (at least one) of program modules 525, such program modules 525 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0114] Bus 530 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0115] The electronic device 500 may also communicate with one or more external devices 600 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 500, and / or may communicate with any device that enables the electronic device 500 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 550. Furthermore, the electronic device 500 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 560. As shown, the network adapter 560 communicates with other modules of the electronic device 500 via a bus 530. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0116] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the exemplary embodiment of the present disclosure.

[0117] The exemplary embodiments of the present disclosure also provide a computer-readable storage medium on which a program product capable of implementing the above-mentioned method of the present specification is stored. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above-mentioned "Exemplary Method" section of the present specification.

[0118] The exemplary embodiments of the present disclosure also provide a program product for implementing the above method, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0119] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0120] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0121] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0122] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0123] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0124] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to exemplary embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0125] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0126] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A prompt word processing method, characterized in that: include: Acquire corpus information, and generate a plurality of first prompt words according to the corpus information; Based on the plurality of first prompt words, determining a plurality of second prompt words; Using a pre-configured prompt word classification model to process the second prompt words, and obtain classification results corresponding to each of the second prompt words; According to the classification results of each of the second prompt words, the plurality of second prompt words are divided into sets to generate a prompt word set.

2. The method according to claim 1, characterized in that The method further comprises: Using the language model to be tested to process a second prompt word in the prompt word set to obtain output data corresponding to the second prompt word; A compliance check is performed on the output data to determine a test result of the language model to be tested according to the compliance check result.

3. The method according to claim 2, characterized in that The performing compliance check on the output data to determine the test result of the language model to be tested according to the compliance check result includes: When it is determined that the compliance rate of the output data is greater than a preset threshold, it is determined that the language model to be tested passes the test.

4. The method according to claim 2, characterized in that: The method further comprises: Acquire a sensitive database, and use the output data to match the sensitive database to determine a sensitive data matching result; According to the sensitive data matching result, the test result of the language model to be tested is checked.

5. The method according to claim 1, characterized in that The step of determining a plurality of second prompt words based on the plurality of first prompt words comprises: Blocking part of the information in the first prompt word to obtain a first middle word; According to the first intermediate word, detecting whether the first prompt word is a negative prompt word; If it is determined that the first prompt word is a negative prompt word, determining a second prompt word according to the first prompt word; If it is determined that the first prompt word is a non-negative prompt word, a second intermediate word is constructed according to the first prompt word, and a second prompt word is determined according to the first prompt word and / or the second intermediate word.

6. The method according to claim 1, characterized in that The method further comprises: Obtaining a first feature extraction model, and performing distillation processing on the first feature extraction model to obtain a second feature extraction model; The prompt word classification model is determined according to the second feature extraction model.

7. The method according to claim 1, characterized in that The classification results include a first classification result and a second classification result: The step of dividing the plurality of second prompt words into sets according to the classification results of the second prompt words to generate a prompt word set includes: The first number of second prompt words of the first classification results and the second number of second prompt words of the second classification results are divided into the same set to generate a prompt word set; the ratio of the first number to the second number is a preset ratio.

8. A prompt word processing device, characterized in that: include: A first prompt word generation module, used to obtain corpus information and generate a plurality of first prompt words according to the corpus information; A second prompt word obtaining module, used for optimizing the first prompt word to obtain a plurality of second prompt words; A second prompt word processing module, used to process the second prompt words using a pre-configured prompt word classification model to obtain classification results corresponding to each of the second prompt words; The prompt word set generation module is used to divide the plurality of second prompt words into sets according to the classification results of each of the second prompt words to generate a prompt word set.

9. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium comprising a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images