Prompt word leakage detection method and device for large language model, equipment and medium

Abstract

The invention discloses a cue word leakage detection method and device for a large language model, equipment and a medium, and relates to the technical field of artificial intelligence, text paragraphs are extracted from user cue words and system cue words, and segmentation, screening, duplicate removal and numbering are performed to obtain a mapping table; segmenting an output result of the large language model to obtain to-be-detected sentences and to-be-detected words; feature words are determined from the first mapping table, a first ratio of the number of the feature words to the number of the words to be detected is calculated, and whether the first ratio is larger than a preset first threshold value or not is judged; if not, comparing the second mapping table and the third mapping table with the to-be-detected sentence to obtain feature sentences, counting the number of the feature sentences, calculating a second ratio of the number to the total number of feature words, and judging whether the second ratio is greater than a preset second threshold value or not; if not, the similarity between the feature sentence and the to-be-detected sentence is calculated, if the similarity is not larger than a preset similarity threshold value, it is judged that no cue word leakage occurs, and the cue word leakage detection performance and efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and medium for detecting prompt word leakage of a large language model. Background Art

[0002] In addition to the method of adding adversarial instructions to the prompt words, the prior art can also use a large model to perform security detection on user behavior in the prompt word content, which can cope with a variety of attacks based on prompt word injection, but it is not specifically aimed at the prompt word leakage problem, and the cost is very high, the performance loss is very large, and training a dedicated large model requires additional costs, and the large model takes a lot of time to classify the user's input and perform multi-angle security detection, especially when the user's input content is very long, the detection may be more time-consuming than answering the user's questions. In order to ensure that the user experience is not significantly reduced, it is necessary to increase the system's computing resources exponentially. And like the method of using adversarial prompt words, another disadvantage of using a large model for detection is the problem of being bypassed. The expression of natural language can become very obscure and tortuous to avoid detection, or it can directly attack the model used for security detection, so that the model outputs the wrong result of "detection passed". The data set used to train the large model is limited, and it is difficult to cover all the attack samples that emerge in an endless stream. The model needs to be retrained after the data set is changed. Just as the virus feature library of antivirus software is updated later than the appearance of the virus itself, the ability of the large model to prevent unknown attacks is insufficient.

[0003] It can be seen from the above that how to accurately identify and determine whether a prompt word leakage occurs, reduce the complexity of prompt word leakage detection, and improve the performance and efficiency of prompt word leakage detection are problems to be solved in the field. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a method, device, equipment and medium for detecting prompt word leakage of a large language model, which can accurately identify and determine whether prompt word leakage occurs, reduce the complexity of prompt word leakage detection, and improve the performance and efficiency of prompt word leakage detection. The specific scheme is as follows:

[0005] In a first aspect, the present application discloses a method for detecting prompt word leakage of a large language model, comprising:

[0006] Obtaining a user prompt word input by a user, and performing key content recognition on the user prompt word and a local system prompt word to obtain key content;

[0007] Extracting text paragraphs marked as prohibited from leakage from the user prompt words and the system prompt words, segmenting, screening, removing duplicates and numbering the text paragraphs according to the key content to obtain a first mapping table of word text-word number, a second mapping table of word number-sentence number, and a third mapping table of sentence number-word number;

[0008] Inputting the user prompt word and the system prompt word into a preset large language model to obtain an output result, and segmenting the output result to obtain sentences to be detected and words to be detected;

[0009] Taking the words in the first mapping table that are consistent with the words to be detected as feature words, calculating a first ratio between the number of the feature words and the number of the words to be detected, and determining whether the first ratio is greater than a preset first threshold;

[0010] If not, comparing the second mapping table and the third mapping table with the sentence to be detected to obtain feature sentences corresponding to the feature words, counting the number of the feature sentences, calculating a second ratio between the number of the feature sentences and the total number of the feature words in the sentence to be detected, and determining whether the second ratio is greater than a preset second threshold;

[0011] If not, the similarity between the feature sentence and the sentence to be detected is calculated; if the similarity is not greater than the preset similarity threshold, it is determined that the output result of the large language model has not suffered from prompt word leakage, so as to complete the detection of the output result without prompt word leakage.

[0012] Optionally, the performing key content recognition on the user prompt word and the local system prompt word to obtain the key content includes:

[0013] Retrieving the pre-written and saved system prompt words from the local computer, identifying the special symbols in the system prompt words, and using the content marked by the special symbols as the key content of the system;

[0014] Key content recognition is performed on the user prompt word to obtain the user's key content.

[0015] Optionally, the segmenting, screening, deduplication and numbering of the text paragraphs according to the key content includes:

[0016] Segmenting the text paragraph into sentences according to the key content;

[0017] Each of the sentences is divided into English words and Chinese characters, and the duplicates are filtered out, and numbers are added to the English words and Chinese characters after the filtering and duplication removal.

[0018] Optionally, segmenting the text paragraph into sentences according to the key content; segmenting each sentence into English words and Chinese characters, includes:

[0019] Using preset symbols to segment the character string and segmenting the text paragraph into sentences according to the key content;

[0020] Each of the sentences is segmented into English words and Chinese characters using a preset English word recognition boundary and a Unicode character set.

[0021] Optionally, the calculating the similarity between the feature sentence and the sentence to be detected includes:

[0022] Screening out a target similarity calculation algorithm from all similarity calculation algorithms; the similarity calculation algorithm includes a Damerau-Levenshtein algorithm, a Euclidean distance algorithm, a cosine similarity algorithm, and a Jacard similarity algorithm;

[0023] The target similarity calculation algorithm is used to calculate the similarity between the feature sentence and the sentence to be detected.

[0024] Optionally, the method for detecting prompt word leakage of a large language model further includes:

[0025] If the similarity between the feature sentence and the sentence to be detected is greater than a preset similarity threshold, then counting the number of the feature words in the current feature sentence;

[0026] If the number of the characteristic words in the current characteristic sentence is greater than a preset threshold, it is determined that the output result of the large language model has prompt word leakage, and the output of the output result is prohibited to complete the detection of the output result with prompt word leakage.

[0027] In a second aspect, the present application discloses a prompt word leakage detection device for a large language model, comprising:

[0028] A key content recognition module is used to obtain a user prompt word input by a user, and perform key content recognition on the user prompt word and a local system prompt word to obtain key content;

[0029] a mapping table determination module, configured to extract text paragraphs marked as prohibited from leakage from the user prompt words and the system prompt words, and segment, screen, remove duplicates, and number the text paragraphs according to the key content to obtain a first mapping table of word text-word number, a second mapping table of word number-sentence number, and a third mapping table of sentence number-word number;

[0030] An output result segmentation module, used for inputting the user prompt word and the system prompt word into a preset large language model to obtain an output result, and segmenting the output result to obtain sentences to be detected and words to be detected;

[0031] A first ratio calculation module, configured to use the words in the first mapping table that are consistent with the words to be detected as feature words, calculate a first ratio between the number of the feature words and the number of the words to be detected, and determine whether the first ratio is greater than a preset first threshold;

[0032] a second ratio calculation module, configured to compare the second mapping table and the third mapping table with the sentence to be detected, if not greater than, to obtain feature sentences corresponding to the feature words, count the number of the feature sentences, calculate a second ratio between the number of the feature sentences and the total number of the feature words in the sentence to be detected, and determine whether the second ratio is greater than a preset second threshold;

[0033] The similarity calculation module is used to calculate the similarity between the feature sentence and the sentence to be detected if it is not greater than, and if the similarity is not greater than a preset similarity threshold, it is determined that the output result of the large language model has not undergone prompt word leakage, so as to complete the detection of the output result without prompt word leakage.

[0034] Optionally, the key content identification module includes:

[0035] A system key content recognition module is used to retrieve the pre-written and saved system prompt words from the local, recognize the special symbols in the system prompt words, and use the content marked by the special symbols as the key content of the system;

[0036] The user key content recognition module is used to perform key content recognition on the user prompt word to obtain the user's key content.

[0037] In a third aspect, the present application discloses an electronic device, comprising:

[0038] Memory, used to store computer programs;

[0039] The processor is used to execute the computer program to implement the aforementioned prompt word leakage detection method for a large language model.

[0040] In a fourth aspect, the present application discloses a computer storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the steps of the aforementioned method for detecting prompt word leakage of a large language model are implemented.

[0041] It can be seen that the present application provides a prompt word leakage detection method for a large language model, including obtaining a user prompt word input by a user, performing key content recognition on the user prompt word and the local system prompt word to obtain the key content; extracting text paragraphs marked as prohibited from leakage from the user prompt word and the system prompt word, segmenting, filtering and deduplicating the text paragraphs according to the key content, and numbering them to obtain a first mapping table of word text-word number, a second mapping table of word number-sentence number, and a third mapping table of sentence number-word number; inputting the user prompt word and the system prompt word into a preset large language model to obtain an output result, segmenting the output result to obtain sentences to be detected and words to be detected; and segmenting the words in the first mapping table that are consistent with the words to be detected. as feature words, calculating a first ratio between the number of the feature words and the number of the words to be detected, and judging whether the first ratio is greater than a preset first threshold value; if not, comparing the second mapping table and the third mapping table with the sentence to be detected to obtain feature sentences corresponding to the feature words, counting the number of the feature sentences, calculating a second ratio between the number of the feature sentences and the total number of the feature words in the sentence to be detected, and judging whether the second ratio is greater than a preset second threshold value; if not, calculating the similarity between the feature sentences and the sentence to be detected, and if the similarity is not greater than the preset similarity threshold value, judging that no prompt word leakage occurs in the output result of the large language model, so as to complete the detection of the output result without prompt word leakage. The present application can identify key content of user prompt words and local system prompt words to obtain key content, extract text paragraphs marked as prohibited leakage from user prompt words and system prompt words, segment, screen and remove duplicates, and number the text paragraphs to obtain a mapping table, use a fast matching feature algorithm to use words in the first mapping table that are consistent with the words to be detected as feature words, calculate a first ratio between the number of feature words and the number of words to be detected, calculate a second ratio between the number of feature sentences and the total number of the feature words in the sentences to be detected, and calculate the similarity between the feature sentences and the sentences to be detected, thereby avoiding the problem of bypassing detection due to fuzzy matching of natural language. If the similarity is not greater than a preset similarity threshold, it is determined that the output result of the large language model has not leaked prompt words, and the output result without prompt word leakage is detected, which solves the problem that the matching accuracy of the large model cannot be quantitatively controlled, and there is no need to train a special large model. It can accurately identify and determine whether prompt word leakage occurs, reduce the complexity of prompt word leakage detection, improve the performance and efficiency of prompt word leakage detection, and occupy less memory. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0043] Figure 1 A flow chart of a method for detecting prompt word leakage of a large language model disclosed in this application;

[0044] Figure 2 A flow chart of segmentation, screening, deduplication and numbering disclosed in this application;

[0045] Figure 3 A flowchart of mapping table comparison and prompt word leakage detection disclosed in this application;

[0046] Figure 4 A schematic diagram of the structure of a prompt word leakage detection device for a large language model disclosed in this application;

[0047] Figure 5 A structural diagram of an electronic device provided for this application. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] In addition to the method of adding adversarial instructions to the prompt words, the prior art can also use a large model to perform security detection on user behavior in the prompt word content, which can cope with a variety of attacks based on prompt word injection, but it is not specifically aimed at the prompt word leakage problem, and the cost is very high, the performance loss is very large, and training a dedicated large model requires additional costs, and the large model takes a lot of time to classify the user's input and perform multi-angle security detection, especially when the user's input content is very long, the detection may be more time-consuming than answering the user's questions. In order to ensure that the user experience is not significantly reduced, it is necessary to increase the system's computing resources exponentially. And like the method of using adversarial prompt words, another disadvantage of using a large model for detection is the problem of being bypassed. The expression of natural language can become very obscure and tortuous to avoid detection, or it can directly attack the model used for security detection, so that the model outputs the wrong result of "detection passed". The data set used to train the large model is limited, and it is difficult to cover all the attack samples that emerge in an endless stream. The model needs to be retrained after the data set is changed. Just as the virus feature library of antivirus software is updated later than the appearance of the virus itself, the ability of the large model to prevent unknown attacks is insufficient. It can be seen from the above that how to accurately identify and determine whether a prompt word leakage occurs, reduce the complexity of prompt word leakage detection, and improve the performance and efficiency of prompt word leakage detection are problems to be solved in the field.

[0050] See also Figure 1 As shown, the embodiment of the present invention discloses a method for detecting prompt word leakage of a large language model, which may specifically include:

[0051] Step S11: obtaining a user prompt word input by a user, and performing key content recognition on the user prompt word and a local system prompt word to obtain key content.

[0052] In this embodiment, a user prompt word input by a user is obtained, a system prompt word pre-written and saved is retrieved locally, and special symbols in the system prompt word are identified, and the content marked with the special symbols is used as the key content of the system; key content identification is performed on the user prompt word to obtain the key content of the user.

[0053] Specifically, prompt words include system prompt words and user prompt words. The creators of the large model system and the intelligent agent are responsible for marking the key content that cannot be leaked in the system prompt words. The input of the large model is generally composed of system prompt words and user prompt words entered by the user. The former is written by the developers of the large model system and the creators of the intelligent agent, and needs to be prevented from being leaked to the user, while the latter generally does not need to be protected. When the creator writes the prompt words, special symbols are used to mark the content that needs to be protected. This special symbol should choose characters that are unlikely to appear in the normal user input, including but not limited to "⋉" and "⋊". The text between a pair of special symbols is the prompt word that needs to be protected.

[0054] Step S12: extract the text paragraphs marked as prohibited from leakage from the user prompt words and the system prompt words, segment, filter, remove duplicates and number the text paragraphs according to the key content to obtain a first mapping table of word text-word number, a second mapping table of word number-sentence number, and a third mapping table of sentence number-word number.

[0055] In this embodiment, text paragraphs marked as prohibited from leakage are extracted from the user prompt words and the system prompt words, and the text paragraphs are divided into sentences according to the key content; each sentence is divided into English words and Chinese characters, and the results are filtered and deduplicated, and numbers are added to the filtered and deduplicated English words and Chinese characters to obtain a first mapping table of word text-word number, a second mapping table of word number-sentence number, and a third mapping table of sentence number-word number.

[0056] The segmentation process is as follows: using preset symbols to segment the character string and segmenting the text paragraph into sentences according to the key content; using preset English word recognition boundaries and Unicode character sets to segment each sentence into English words and Chinese characters.

[0057] Specifically, first extract all the prompt word paragraphs marked in step S11 from the user prompt words and the system prompt words, then segment the paragraph into sentences, and segment each sentence into English words and Chinese characters, and then assign numbers to each word, Chinese character and sentence after screening and deduplication, and obtain three mapping tables, thereby forming a feature set. The process of segmentation, screening, deduplication and numbering is as follows: Figure 2Shown as follows: The algorithm for splitting a text paragraph into sentences is to split the string using symbols such as full stops, exclamation marks, question marks, etc. that can mark the end of a sentence, and exclude cases where an English full stop is used as a decimal point (e.g., 3.14) and abbreviated letters (e.g., e.g.). The algorithm for splitting a sentence into English words and Chinese characters is to identify English words based on the boundaries of blanks and non-English letters, and identify Chinese characters within the range of the Unicode Basic Chinese Character Set 4E00-9FA5. English words with fewer than 4 letters, particularly commonly used Chinese characters (such as "的" and "个"), and other characters not within the recognition range are ignored. Selectively ignoring some words can improve the detection efficiency, but it is not necessary. The English words and Chinese characters (hereinafter referred to as words) obtained by screening and deduplication through the above algorithms are assigned different numbers, and each sentence is also assigned a different number. The words and sentences are numbered separately, both starting from 1. The numerical values between the two numberings can be repeated. Then, three mapping tables are constructed: The first mapping table is the mapping from "word text to word number", where one word corresponds to only one number; the second mapping table is the mapping from "word number to sentence number", which is a one-to-many mapping, indicating that one word appears in several sentences; the third mapping table is the mapping from "sentence number to word number", which is also a one-to-many mapping, indicating that one sentence contains several words.

[0058] For example, for a passage like "Where are you from? I am from China.", it contains two sentences. After processing the first sentence, two words, Where and from, are obtained, numbered 1 and 2 respectively, and the sentence itself is numbered 1. After processing the second sentence, two words, from and China, are obtained, numbered 2 and 3 respectively, and the sentence itself is numbered 2. Among them, the word from, since it appears repeatedly, only occupies one number 2. Then, the content of the first mapping table is {"Where"→1, "from"→2, "China"→3}, the content of the second mapping table is {1→1, 2→[1,2], 3→2}, and the content of the third mapping table is {1→[1,2], 2→[2,3]}. Here, the arrow → represents a mapping from the left to the right, and the square brackets [] represent a set of numbers. The three mapping tables form a feature set. The words in the table are called feature words, and the sentences in the table are called feature sentences.

[0059] In addition, to identify the content that needs to be protected in the prompt words, in addition to using special symbols, the method of recording the offset and length of the content in the entire prompt word can also be used. Splitting sentences and words can also be achieved through features such as the capitalization of the first letter of an English word and the blank after a full stop, and the end of a sentence can also be recognized. In addition to splitting Chinese sentences into individual characters using the Chinese character set, word segmentation can also be achieved using various schemes such as dictionaries.

[0060] Step S13: input the user prompt word and the system prompt word into a preset large language model to obtain an output result, and segment the output result to obtain sentences and words to be detected.

[0061] In this embodiment, the output result of the large model is used as the text to be processed and segmented into sentences and words. The segmentation algorithm is consistent with the description in the above steps. Considering that the large model always outputs word by word, rather than outputting a complete answer at one time, for scenarios where it is impossible to wait for the entire output to be completed before testing, the output content needs to be cached until the end of a sentence. In this case, each text to be processed contains only one sentence. Finally, the sentences to be tested and the words to be tested are obtained.

[0062] Step S14: taking the words in the first mapping table that are consistent with the words to be detected as feature words, calculating a first ratio between the number of the feature words and the number of the words to be detected, and determining whether the first ratio is greater than a preset first threshold.

[0063] In this embodiment, each sentence to be detected and each word to be detected are processed. The number of each word is obtained by querying the first mapping table. If the word is not in the first mapping table, its number is 0. The number of each word in this sentence is recorded, and the words with non-0 numbers are characteristic words. The ratio of the number of characteristic words to the total number of words in this sentence is calculated. If the ratio exceeds the preset first threshold (for example, 0.8), the detection is continued, otherwise the next sentence is processed from the beginning.

[0064] Step S15: If it is not greater than, then compare the second mapping table and the third mapping table with the sentence to be detected to obtain feature sentences corresponding to the feature words, count the number of the feature sentences, calculate the second ratio between the number of the feature sentences and the total number of the feature words in the sentence to be detected, and determine whether the second ratio is greater than a preset second threshold.

[0065] In this embodiment, each characteristic word of the sentence to be processed belongs to which characteristic sentences by querying the second mapping table. A word may appear in multiple sentences. The number of times each characteristic sentence number appears is accumulated. For sentence numbers whose number of appearances is not 0, all characteristic word numbers and the total number of characteristic words (i.e., the number of numbers) contained in this characteristic sentence are obtained by querying the third mapping table. The ratio of the number of times the characteristic sentence number appears to the total number of characteristic words in the sentence is calculated. If the ratio exceeds the preset second threshold (e.g., 0.8), continue to detect, otherwise start processing the next sentence from the beginning.

[0066] Step S16: If it is not greater than, then calculate the similarity between the feature sentence and the sentence to be detected. If the similarity is not greater than the preset similarity threshold, then determine that the output result of the large language model has not undergone prompt word leakage, so as to complete the detection of the output result without prompt word leakage.

[0067] In this embodiment, if the second ratio is not greater than a preset second threshold, a target similarity calculation algorithm is screened out from all similarity calculation algorithms; the similarity calculation algorithm includes a Damerau-Levenshtein algorithm, a Euclidean distance algorithm, a cosine similarity algorithm, and a Jacard similarity algorithm; the target similarity calculation algorithm is used to calculate the similarity between the feature sentence and the sentence to be detected, and if the similarity is not greater than a preset similarity threshold, it is determined that no prompt word leakage occurs in the output result of the large language model, so as to complete the detection of the output result without prompt word leakage.

[0068] In addition, if the similarity between the feature sentence and the sentence to be detected is greater than a preset similarity threshold, the number of the feature words in the current feature sentence is counted; if the number of the feature words in the current feature sentence is greater than a preset threshold, it is determined that the output result of the large language model has prompt word leakage, and the output of the output result is prohibited to complete the detection of the output result in which prompt word leakage has occurred.

[0069] Specifically, the prompt word leakage detection process is as follows: Figure 3 As shown, each feature sentence whose proportion exceeds the threshold is calculated in turn with the sentence to be processed for similarity. There are many options for algorithms to calculate similarity. For example, the Damerau-Levenshtein algorithm can be used. In order to improve the execution efficiency of the algorithm and reduce memory usage, the algorithm can be modified. The original algorithm calculates the string, and the modified algorithm calculates the word number sequence. That is, the object of comparison of the modified algorithm is the word number sequence contained in the feature sentence and the word number (including number 0) sequence contained in the sentence to be processed. The value range of the calculation result is 0 to 1, and 1 means that the two comparison objects are exactly the same. If the similarity exceeds the preset similarity threshold (for example, 0.8), continue to detect, otherwise start processing the next sentence from the beginning.

[0070] Finally, for the remaining sentences to be processed after the above screening, the number of characteristic words contained in these sentences is accumulated. If the total number of characteristic words exceeds the preset threshold, it is considered that the prompt word leakage occurs, and all text output of this large model is prohibited. The preset threshold here is related to the size of the feature set. For example, based on the average number of words in all characteristic sentences, the preset threshold is set to 0.8 times the average number of words.

[0071] The innovation of this application is that by testing the output of the large model instead of the input, it is possible to accurately determine whether a prompt word leak has occurred. Regardless of the means of attack, only the result is considered to be problematic, so unknown attack methods can be prevented; a text similarity algorithm is used instead of a large language model, and the algorithm processes word encodings, so there is no problem of bypassing detection due to fuzzy matching of natural language. In addition, each step of the algorithm has an adjustable threshold, which can be adjusted between completely accurate matching and a certain allowable error, while the matching accuracy of the large model cannot be quantitatively controlled; a feature set is dynamically constructed based on the input content of the large model, without the need to train a special large model. This feature set is highly targeted, takes up little memory, and has a simple algorithm, so the detection performance is very high and the cost is very low.

[0072] In this embodiment, a user prompt word input by a user is obtained, and the key content of the user prompt word and the local system prompt word is identified to obtain the key content; a text paragraph marked as prohibited from leakage is extracted from the user prompt word and the system prompt word, and the text paragraph is segmented, filtered to remove duplicates and numbered according to the key content to obtain a first mapping table of word text-word number, a second mapping table of word number-sentence number, and a third mapping table of sentence number-word number; the user prompt word and the system prompt word are input into a preset large language model to obtain an output result, and the output result is segmented to obtain sentences to be detected and words to be detected; the words in the first mapping table that are consistent with the words to be detected are used as feature words, and the A first ratio between the number of characteristic words and the number of the words to be detected, and judging whether the first ratio is greater than a preset first threshold value; if not, comparing the second mapping table and the third mapping table with the sentence to be detected to obtain characteristic sentences corresponding to the characteristic words, counting the number of the characteristic sentences, calculating a second ratio between the number of the characteristic sentences and the total number of the characteristic words in the sentence to be detected, and judging whether the second ratio is greater than a preset second threshold value; if not, calculating the similarity between the characteristic sentence and the sentence to be detected, and if the similarity is not greater than the preset similarity threshold value, judging that the output result of the large language model has not undergone prompt word leakage, so as to complete the detection of the output result without prompt word leakage. The present application can identify key content of user prompt words and local system prompt words to obtain key content, extract text paragraphs marked as prohibited leakage from user prompt words and system prompt words, segment, screen and remove duplicates, and number the text paragraphs to obtain a mapping table, use a fast matching feature algorithm to use words in the first mapping table that are consistent with the words to be detected as feature words, calculate a first ratio between the number of feature words and the number of words to be detected, calculate a second ratio between the number of feature sentences and the total number of the feature words in the sentences to be detected, and calculate the similarity between the feature sentences and the sentences to be detected, thereby avoiding the problem of bypassing detection due to fuzzy matching of natural language. If the similarity is not greater than a preset similarity threshold, it is determined that the output result of the large language model has not leaked prompt words, and the output result without prompt word leakage is detected, which solves the problem that the matching accuracy of the large model cannot be quantitatively controlled, and there is no need to train a special large model. It can accurately identify and determine whether prompt word leakage occurs, reduce the complexity of prompt word leakage detection, improve the performance and efficiency of prompt word leakage detection, and occupy less memory.

[0073] See also Figure 4 As shown, the embodiment of the present invention discloses a prompt word leakage detection device for a large language model, which may specifically include:

[0074] The key content recognition module 11 is used to obtain the user prompt word input by the user, and perform key content recognition on the user prompt word and the local system prompt word to obtain the key content;

[0075] A mapping table determination module 12 is used to extract text paragraphs marked as prohibited from leakage from the user prompt words and the system prompt words, and segment, filter, remove duplicates and number the text paragraphs according to the key content to obtain a first mapping table of word text-word number, a second mapping table of word number-sentence number, and a third mapping table of sentence number-word number;

[0076] An output result segmentation module 13, used for inputting the user prompt word and the system prompt word into a preset large language model to obtain an output result, and segmenting the output result to obtain sentences to be detected and words to be detected;

[0077] A first ratio calculation module 14 is used to use the words in the first mapping table that are consistent with the words to be detected as feature words, calculate a first ratio between the number of the feature words and the number of the words to be detected, and determine whether the first ratio is greater than a preset first threshold;

[0078] A second ratio calculation module 15 is used for comparing the second mapping table and the third mapping table with the sentence to be detected to obtain a feature sentence corresponding to the feature word, counting the number of the feature sentences, calculating a second ratio between the number of the feature sentences and the total number of the feature words in the sentence to be detected, and determining whether the second ratio is greater than a preset second threshold;

[0079] The similarity calculation module 16 is used to calculate the similarity between the feature sentence and the sentence to be detected if it is not greater than, and if the similarity is not greater than the preset similarity threshold, it is determined that the output result of the large language model has not undergone prompt word leakage, so as to complete the detection of the output result without prompt word leakage.

[0080] In this embodiment, a user prompt word input by a user is obtained, and the key content of the user prompt word and the local system prompt word is identified to obtain the key content; a text paragraph marked as prohibited from leakage is extracted from the user prompt word and the system prompt word, and the text paragraph is segmented, filtered to remove duplicates and numbered according to the key content to obtain a first mapping table of word text-word number, a second mapping table of word number-sentence number, and a third mapping table of sentence number-word number; the user prompt word and the system prompt word are input into a preset large language model to obtain an output result, and the output result is segmented to obtain sentences to be detected and words to be detected; the words in the first mapping table that are consistent with the words to be detected are used as feature words, and the A first ratio between the number of characteristic words and the number of the words to be detected, and judging whether the first ratio is greater than a preset first threshold value; if not, comparing the second mapping table and the third mapping table with the sentence to be detected to obtain characteristic sentences corresponding to the characteristic words, counting the number of the characteristic sentences, calculating a second ratio between the number of the characteristic sentences and the total number of the characteristic words in the sentence to be detected, and judging whether the second ratio is greater than a preset second threshold value; if not, calculating the similarity between the characteristic sentence and the sentence to be detected, and if the similarity is not greater than the preset similarity threshold value, judging that the output result of the large language model has not undergone prompt word leakage, so as to complete the detection of the output result without prompt word leakage. The present application can identify key content of user prompt words and local system prompt words to obtain key content, extract text paragraphs marked as prohibited leakage from user prompt words and system prompt words, segment, screen and remove duplicates, and number the text paragraphs to obtain a mapping table, use a fast matching feature algorithm to use words in the first mapping table that are consistent with the words to be detected as feature words, calculate a first ratio between the number of feature words and the number of words to be detected, calculate a second ratio between the number of feature sentences and the total number of the feature words in the sentences to be detected, and calculate the similarity between the feature sentences and the sentences to be detected, thereby avoiding the problem of bypassing detection due to fuzzy matching of natural language. If the similarity is not greater than a preset similarity threshold, it is determined that the output result of the large language model has not leaked prompt words, and the output result without prompt word leakage is detected, which solves the problem that the matching accuracy of the large model cannot be quantitatively controlled, and there is no need to train a special large model. It can accurately identify and determine whether prompt word leakage occurs, reduce the complexity of prompt word leakage detection, improve the performance and efficiency of prompt word leakage detection, and occupy less memory.

[0081] In some specific embodiments, the key content identification module 11 may specifically include:

[0082] A system key content recognition module is used to retrieve the pre-written and saved system prompt words from the local, recognize the special symbols in the system prompt words, and use the content marked by the special symbols as the key content of the system;

[0083] The user key content recognition module is used to perform key content recognition on the user prompt word to obtain the user's key content.

[0084] In some specific embodiments, the mapping table determination module 12 may specifically include:

[0085] A content segmentation module, used to segment the text paragraph into sentences according to the key content;

[0086] The screening and de-duplication module is used to divide each sentence into English words and Chinese characters, and screen and de-duplication them, and add numbers to the English words and Chinese characters after screening and de-duplication.

[0087] In some specific embodiments, the mapping table determination module 12 may specifically include:

[0088] A sentence segmentation module, used to segment the character string using preset symbols and to segment the text paragraph into sentences according to the key content;

[0089] The word and Chinese character segmentation module is used to segment each sentence into English words and Chinese characters using a preset English word recognition boundary and a Unicode character set.

[0090] In some specific embodiments, the similarity calculation module 16 may specifically include:

[0091] An algorithm screening module, used to screen out a target similarity calculation algorithm from all similarity calculation algorithms; the similarity calculation algorithms include Damerau-Levenshtein algorithm, Euclidean distance algorithm, cosine similarity algorithm and Jacard similarity algorithm;

[0092] The similarity calculation module is used to calculate the similarity between the feature sentence and the sentence to be detected by using the target similarity calculation algorithm.

[0093] In some specific embodiments, the prompt word leakage detection device of the large language model may further include:

[0094] A quantity statistics module, used for counting the number of the characteristic words in the current characteristic sentence if the similarity between the characteristic sentence and the sentence to be detected is greater than a preset similarity threshold;

[0095] The prompt word leakage module is used to determine that the output result of the large language model has prompt word leakage if the number of the characteristic words in the current characteristic sentence is greater than a preset threshold, and prohibit outputting the output result to complete the detection of the output result with prompt word leakage.

[0096] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the method for detecting the leakage of a large language model by an electronic device disclosed in any of the aforementioned embodiments.

[0097] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0098] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 221, a computer program 222 and data 223, etc. The storage method can be temporary storage or permanent storage.

[0099] Among them, the operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20 to realize the operation and processing of the data 223 in the memory 22 by the processor 21, which can be Windows, Unix, Linux, etc. In addition to including a computer program that can be used to complete the prompt word leakage detection method of the large language model performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks. In addition to including data transmitted from an external device received by the prompt word leakage detection device of the large language model, the data 223 can also include data collected by its own input and output interface 25, etc.

[0100] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0101] Furthermore, an embodiment of the present application also discloses a computer-readable storage medium, in which a computer program is stored. When the computer program is loaded and executed by a processor, the steps of the prompt word leakage detection method for a large language model disclosed in any of the aforementioned embodiments are implemented.

[0102] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0103] The above is a detailed introduction to the prompt word leakage detection method, device, equipment and storage medium of a large language model provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for detecting prompt word leakage of a large language model, characterized in that: include: Obtaining a user prompt word input by a user, and performing key content recognition on the user prompt word and a local system prompt word to obtain key content; Extracting text paragraphs marked as prohibited from leakage from the user prompt words and the system prompt words, segmenting, screening, removing duplicates and numbering the text paragraphs according to the key content to obtain a first mapping table of word text-word number, a second mapping table of word number-sentence number, and a third mapping table of sentence number-word number; Inputting the user prompt word and the system prompt word into a preset large language model to obtain an output result, and segmenting the output result to obtain sentences to be detected and words to be detected; Taking the words in the first mapping table that are consistent with the words to be detected as feature words, calculating a first ratio between the number of the feature words and the number of the words to be detected, and determining whether the first ratio is greater than a preset first threshold; If not, comparing the second mapping table and the third mapping table with the sentence to be detected to obtain feature sentences corresponding to the feature words, counting the number of the feature sentences, calculating a second ratio between the number of the feature sentences and the total number of the feature words in the sentence to be detected, and determining whether the second ratio is greater than a preset second threshold; If not, the similarity between the feature sentence and the sentence to be detected is calculated; if the similarity is not greater than the preset similarity threshold, it is determined that the output result of the large language model has not suffered from prompt word leakage, so as to complete the detection of the output result without prompt word leakage.

2. The method for detecting prompt word leakage of a large language model according to claim 1, characterized in that: The performing key content recognition on the user prompt word and the local system prompt word to obtain the key content includes: Retrieving the pre-written and saved system prompt words from the local computer, identifying the special symbols in the system prompt words, and using the content marked by the special symbols as the key content of the system; Key content recognition is performed on the user prompt word to obtain the user's key content.

3. The method for detecting prompt word leakage of a large language model according to claim 1, characterized in that: The segmenting, screening, deduplication and numbering of the text paragraphs according to the key content includes: Segmenting the text paragraph into sentences according to the key content; Each of the sentences is divided into English words and Chinese characters, and the duplicates are filtered out, and numbers are added to the English words and Chinese characters after the filtering and duplication removal.

4. The method for detecting prompt word leakage of a large language model according to claim 3, characterized in that: said dividing the text paragraph into sentences according to the key content; Segment each of the sentences into English words and Chinese characters, including: Using preset symbols to segment the character string and segmenting the text paragraph into sentences according to the key content; Each of the sentences is segmented into English words and Chinese characters using a preset English word recognition boundary and a Unicode character set.

5. The method for detecting prompt word leakage of a large language model according to claim 1, characterized in that: The calculating the similarity between the feature sentence and the sentence to be detected includes: Screening out a target similarity calculation algorithm from all similarity calculation algorithms; the similarity calculation algorithm includes a Damerau-Levenshtein algorithm, a Euclidean distance algorithm, a cosine similarity algorithm, and a Jacard similarity algorithm; The target similarity calculation algorithm is used to calculate the similarity between the feature sentence and the sentence to be detected.

6. The method for detecting prompt word leakage of a large language model according to any one of claims 1 to 5, characterized in that: Also includes: If the similarity between the feature sentence and the sentence to be detected is greater than a preset similarity threshold, then counting the number of the feature words in the current feature sentence; If the number of the characteristic words in the current characteristic sentence is greater than a preset threshold, it is determined that the output result of the large language model has prompt word leakage, and the output of the output result is prohibited to complete the detection of the output result with prompt word leakage.

7. A prompt word leakage detection device for a large language model, characterized in that: include: A key content recognition module is used to obtain a user prompt word input by a user, and perform key content recognition on the user prompt word and a local system prompt word to obtain key content; a mapping table determination module, configured to extract text paragraphs marked as prohibited from leakage from the user prompt words and the system prompt words, and segment, screen, remove duplicates, and number the text paragraphs according to the key content to obtain a first mapping table of word text-word number, a second mapping table of word number-sentence number, and a third mapping table of sentence number-word number; An output result segmentation module, used for inputting the user prompt word and the system prompt word into a preset large language model to obtain an output result, and segmenting the output result to obtain sentences to be detected and words to be detected; A first ratio calculation module, configured to use the words in the first mapping table that are consistent with the words to be detected as feature words, calculate a first ratio between the number of the feature words and the number of the words to be detected, and determine whether the first ratio is greater than a preset first threshold; a second ratio calculation module, configured to compare the second mapping table and the third mapping table with the sentence to be detected, if not greater than, to obtain feature sentences corresponding to the feature words, count the number of the feature sentences, calculate a second ratio between the number of the feature sentences and the total number of the feature words in the sentence to be detected, and determine whether the second ratio is greater than a preset second threshold; The similarity calculation module is used to calculate the similarity between the feature sentence and the sentence to be detected if it is not greater than, and if the similarity is not greater than a preset similarity threshold, it is determined that the output result of the large language model has not undergone prompt word leakage, so as to complete the detection of the output result without prompt word leakage.

8. The device for detecting leakage of prompt words of a large language model according to claim 7, characterized in that: The key content identification module includes: A system key content recognition module is used to retrieve the pre-written and saved system prompt words from the local, recognize the special symbols in the system prompt words, and use the content marked by the special symbols as the key content of the system; The user key content recognition module is used to perform key content recognition on the user prompt word to obtain the user's key content.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor is used to execute the computer program to implement the prompt word leakage detection method of a large language model as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: Used to store a computer program; wherein, when the computer program is executed by a processor, the prompt word leakage detection method for a large language model as described in any one of claims 1 to 6 is implemented.

Images