Prompt word attack detection method and system of large language model and electronic equipment

Abstract

The invention provides a cue word attack detection method and system of a large language model and electronic equipment, and relates to the technical field of text process.The method comprises the steps that a to-be-processed text block in an original text is determined; performing heuristic semantic analysis on the text block to obtain a first suspicious text block and a first risk score thereof; performing semantic analysis on the first suspicious text block and the context information thereof through a first language model to obtain a second suspicious text block and a second risk score thereof; performing structured verification of the attack intention on the second suspicious text block according to a preset meta prompt word through a second language model to obtain an attack intention verification result; the attack intention verification result comprises a third suspicious text block and a third risk score thereof; and executing a target response action on the original text based on the first risk score, the second risk score and the third risk score. According to the method, the efficient processing requirement of the long text can be considered, and the detection accuracy of complex attacks can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of text processing, and in particular to a prompt word attack detection method and system for a large language model and an electronic device. BACKGROUND

[0002] With the wide application of large language models (LLM) in intelligent customer service, content generation and other fields, prompt word injection attacks have become a serious security threat. Attackers can hijack the expected functions of large language models by embedding malicious instructions in input text, and then perform unauthorized operations, which poses a serious security problem.

[0003] To deal with prompt word injection attacks, some related technologies use a filtering method based on rules and keywords. This method has very weak defense capabilities, and attackers can easily bypass it by simply replacing synonyms, character encoding or text obfuscation, resulting in a very high false negative rate. Some related technologies use a detection method based on a single-stage machine learning model. This method uses a machine learning model, which introduces significant computational delay, and is prone to failure in long text due to dilution of malicious signals, also resulting in a very high false negative rate.

[0004] Therefore, the current detection schemes for prompt word injection attacks generally have the problems of low detection accuracy and insufficient processing of long text. SUMMARY

[0005] Therefore, the present application provides a prompt word attack detection method and system for a large language model, which can balance the efficient processing of long text and improve the detection accuracy of complex attacks.

[0006] In a first aspect, an embodiment of the present application provides a prompt word attack detection method for a large language model, the method comprising the following steps: determining a text block to be processed in an original text; performing heuristic semantic analysis on the text block to obtain a first suspicious text block and a first risk score of the first suspicious text block; performing semantic analysis on the first suspicious text block and its context information through a pre-trained first language model to obtain a second suspicious text block and a second risk score of the second suspicious text block; performing structured verification of attack intent on the second suspicious text block according to a preset meta-prompt word through a pre-trained second language model to obtain an attack intent verification result; wherein the attack intent verification result includes a third suspicious text block and a third risk score of the third suspicious text block; perform a target response action on the original text based on the first risk score, the second risk score and the third risk score.

[0007] In some embodiments, the determining of the text block to be processed in the original text comprises: performing a standardization process on the original text to obtain a target text; if the length of the target text is less than a preset length threshold, determining the target text as a text block; if the length of the target text is not less than the preset length threshold, performing semantic segmentation on the target text to obtain the text block.

[0008] In some embodiments, the performing of the semantic segmentation on the target text to obtain the text block comprises: generating vector embeddings for each sentence in the target text through a preset sentence converter model; determining a cosine distance between the vector embeddings corresponding to each pair of adjacent sentences; determining a sentence boundary at which a semantic change occurs according to the cosine distance; segmenting the target text into text blocks based on the sentence boundary; wherein adjacent text blocks include an overlapping text of a first preset length.

[0009] In some embodiments, the heuristic semantic analysis of the text block to obtain a first suspicious text block and a first risk score of the first suspicious text block comprises: performing semantic matching on the text block using a preset regular expression library to obtain a first suspicious text block; wherein the regular expression library includes multiple types of patterns related to prompt word injection; and / or, performing semantic matching on the text block according to a preset keyword dictionary to obtain the first suspicious text block; wherein the keyword dictionary includes keywords related to attack behavior; and / or, calculating an information entropy of the text block, and determining a text block with an information entropy higher than a preset information entropy as the first suspicious text block; determining a first risk score for the first suspicious text block.

[0010] In some embodiments, the performing of the semantic analysis on the first suspicious text block and its context information through a pre-trained first language model to obtain a second suspicious text block and a second risk score of the second suspicious text block comprises: obtaining a pre-trained first language model; wherein the first language model is a model based on a Transformer architecture and is trained using prompt word samples labeled with classification labels; inputting the first suspicious text block and context information thereof into the first language model; wherein the context information of the first suspicious text block comprises: position information of the first suspicious text block in the original text, and overlapping text of a second preset length between the first suspicious text block and adjacent text blocks; determining, by the first language model, a target classification label corresponding to the first suspicious text block and a second risk score of the target classification label according to the first suspicious text block and context information thereof; determining, as a second suspicious text block, the first suspicious text block with a second risk score higher than a preset score value.

[0011] In some embodiments, the structured verification of attack intent of the second suspicious text block and context information thereof according to the preset meta-prompt word by the pre-trained second language model obtains an attack intent verification result, comprising: obtaining a pre-trained second language model; wherein the second language model is a general large language model; inputting a preset meta-prompt word, the second suspicious text block and context information thereof into the second language model; wherein the meta-prompt word comprises a guiding question; performing, by the second language model, structured verification of attack intent of the second suspicious text block according to the meta-prompt word, the guiding question, the second suspicious text block and context information thereof, to obtain an attack intent verification result; wherein the attack intent verification result comprises: a third suspicious text block, a third risk score of the third suspicious text block, and a ruling reason of the third suspicious text block.

[0012] In some embodiments, the target response action is performed on the original text based on the first risk score, the second risk score and the third risk score, comprising: performing weighted scoring on the first risk score, the second risk score and the third risk score to obtain a target risk score of the original text; determining a target response action corresponding to the target risk score according to a corresponding relationship between risk scores and response actions; performing the target response action on the original text.

[0013] In some embodiments, before determining the text block to be processed in the original text, the method further comprises: intercepting a text detection request of a user terminal in response to the text detection request, and obtaining an original text to be subjected to prompt word attack detection.

[0014] In a second aspect, an embodiment of the present application provides a prompt word attack detection system of a large language model, the system comprising the following modules: a text block determination module configured to determine a text block to be processed in the original text; wherein the text block is marked with position information; a first semantic analysis module configured to perform heuristic semantic analysis on the text block to obtain a first suspicious text block and a first risk score of the first suspicious text block; a second semantic analysis module configured to perform semantic analysis on the first suspicious text block and context information thereof by using a pre-trained first language model to obtain a second suspicious text block and a second risk score of the second suspicious text block; an attack intent verification module configured to perform structured verification of an attack intent of the second suspicious text block according to a preset meta prompt word by using a pre-trained second language model to obtain an attack intent verification result; wherein the attack intent verification result comprises a third suspicious text block and a third risk score of the third suspicious text block; an action response module configured to perform a target response action on the original text based on the first risk score, the second risk score, and the third risk score.

[0015] In a third aspect, an embodiment of the present application further provides an electronic device comprising a memory and a processor, and the memory stores a computer program capable of running on the processor, wherein the processor executes the computer program to implement the steps of the prompt word attack detection method of the large language model according to the first aspect.

[0016] The embodiments of the present application bring at least the following beneficial effects: The present application provides a prompt word attack detection method, system and electronic device of a large language model, the method comprising: determining a text block to be processed in an original text; performing heuristic semantic analysis on the text block to obtain a first suspicious text block and a first risk score of the first suspicious text block; performing semantic analysis on the first suspicious text block and context information thereof by using a pre-trained first language model to obtain a second suspicious text block and a second risk score of the second suspicious text block; performing structured verification of an attack intent of the second suspicious text block according to a preset meta prompt word by using a pre-trained second language model to obtain an attack intent verification result; wherein the attack intent verification result comprises a third suspicious text block and a third risk score of the third suspicious text block; and performing a target response action on the original text based on the first risk score, the second risk score, and the third risk score.

[0017] The technical scheme can realize full-process coverage from rapid preliminary screening to deep semantic analysis and attack intention verification through constructing a three-level progressive detection method. Firstly, heuristic semantic analysis is used as the preliminary screening link, which can complete the preliminary filtering of large-scale text blocks with extremely low cost, effectively guarantee the system basic throughput performance, greatly reduce the calculation cost of subsequent deep analysis, and guarantee the processing efficiency. Secondly, the first language model performs context-aware semantic analysis on the first suspicious text block, which can identify medium complexity attacks disguised by language, accurately filter the second suspicious text block with high risk, and improve the detection accuracy. Finally, the second language model is used to perform structured verification of attack intention combined with meta-prompt words, so that complex hidden attacks are not missed, and accurate identification and interpretable decision of new and complex attack intentions are realized. The above multi-stage cooperative detection scheme forms a defense-in-depth system, which not only maintains the real-time performance of the system through the front-end rapid filtering, but also ensures the detection accuracy through the back-end deep analysis, so that single-dimensional misjudgment can be avoided, and the target response action is more scientific. Therefore, the scheme can not only meet the efficient processing needs of long text, but also improve the detection accuracy of complex attacks, and provides reliable protection for the safe application of large language models.

[0018] Other features and advantages of the present application will be set forth in the following description, or will be apparent from the description, or can be learned by practice of the application according to the above technical solutions.

[0019] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the following preferred embodiments are described in detail below, and the accompanying drawings are described as follows. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0021] Figure 1 A flowchart of a prompt word attack detection method of a large language model provided by an embodiment of the present application; Figure 2 A work flow diagram of a prompt word attack detection method of a large language model provided by an embodiment of the present application; Figure 3 A structural diagram of a prompt word attack detection system of a large language model provided by an embodiment of the present application; Figure 4 A structural diagram of an electronic device provided by an embodiment of the present application.

[0022] icon: 301 - text block determination module; 302 - first semantic analysis module; 303 - second semantic analysis module; 304 - attack intent verification module; 305 - action response module; 101 - processor; 102 - memory; 103 - bus; 104 - communication interface. DETAILED DESCRIPTION

[0023] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the present application will be described clearly and completely below with reference to the drawings. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0024] With the wide deployment and application of large language models in intelligent customer service, content creation assistants, code generation tools, and other fields, prompt injection attacks have become a serious security threat. Attackers embed malicious instructions in input text to hijack the intended functions of large language models, and then perform unauthorized operations, such as leaking sensitive data, bypassing security policies, generating harmful content, or obtaining improper access to backend systems.

[0025] To deal with prompt injection attacks, some related technologies use a filtering method based on rules and keywords. This method relies on pre-defined regular expression patterns for detecting suspicious syntax, keyword blacklists (such as "ignore previous instructions", "forget your identity", etc.), and simple heuristic rules (such as excessive punctuation, abnormal character sequences, or code blocks) to identify malicious inputs. However, the defense capability of this method is very weak, and attackers can easily bypass detection by using text obfuscation techniques or by changing the wording, resulting in a very high false negative rate. The above text obfuscation techniques such as using Base64 encoding, homonym replacement, inserting zero-width characters, etc.; the above wording changes such as changing "ignore previous instructions" to "your previous instructions are void, now follow my new arrangements". Moreover, this method cannot understand the real semantic intent behind the words, resulting in its failure when facing slightly changed attacks.

[0026] Some related technologies use a detection method based on a single-stage machine learning model. This method trains a single machine learning model, such as a fine-tuned BERT model or other classifiers based on the Transformer architecture, to make an end-to-end maliciousity judgment on the entire input prompt by the machine learning model, and output the classification result of being benign or malicious. However, this type of method has the following technical defects: (1) Performance bottleneck problem: For application scenarios that require real-time interaction, each input, especially long text input, is analyzed by a machine learning model, which introduces significant computational delay. This high delay is unacceptable when handling large-scale concurrent requests, and cannot meet the performance requirements of real-time interaction scenarios, severely limiting the practical application value of this solution in production environments.

[0027] (2) Detection failure problem in long text context: When malicious instructions are cleverly embedded in a long and normal content document (for example, inserting a malicious instruction in a 10,000-word technical report), the malicious signal is severely diluted on the macro scale of the entire text. When processing the entire text, a single machine learning model will have its attention scattered by a large amount of harmless content, making it difficult to capture this localized and weak malicious signal, which can be compared to "finding a needle in a haystack", resulting in a high false negative rate and a significant decrease in effectiveness when processing long text input.

[0028] (3) Lack of detection granularity and explainability: This method usually only provides binary judgment results of benign or malicious, and cannot explain the reason why the input is marked as malicious, nor can it accurately locate the specific malicious segment in the text. This makes subsequent risk analysis, security audit, and automatic repair extremely difficult.

[0029] Therefore, the above detection scheme for prompt injection attack cannot effectively balance the accuracy and computational performance of detection. Especially in the increasingly common long text input scenario, this contradiction becomes more and more prominent. Simple filtering schemes are fast but lack depth of analysis and are easily bypassed; while complex machine learning models have strong analysis capabilities but are slow and are easily ineffective in long text due to signal dilution.

[0030] Related solutions generally use a "one-size-fits-all" single-stage processing mode, either performing shallow scanning on all text content or performing deep analysis on all text content, both of which have inherent defects. The above problems seriously hinder the secure deployment of large language models in the application of complex documents such as legal contract review, financial report analysis, and technical document summary, which are exactly the areas most affected by advanced prompt injection attacks.

[0031] In this case, in order to better deal with prompt injection attacks and improve the detection accuracy of prompt injection attacks and improve the ability to handle long text, the present disclosure provides a prompt attack detection method, system and electronic device for a large language model.

[0032] For the convenience of understanding the present embodiment, first, a large language model prompt word attack detection method disclosed by the present embodiment is introduced in detail, which is applied to the scene of detecting prompt word attacks in large language model applications. As shown in Figure 1 The large language model prompt word attack detection method can include the following steps.

[0033] Step S101, determine the text block to be processed in the original text; wherein the text block is marked with position information.

[0034] In the present embodiment, the user can input an original text of any length, such as technical documents, legal contracts, and financial reports, etc. At this time, the user end initiates a text detection request carrying the original text; the system for implementing the large language model prompt word attack detection method responds to the text detection request of the user end, intercepts the text detection request, and obtains the original text to be subjected to prompt word attack detection.

[0035] A series of standardized cleaning operations are performed on the original text to obtain the target text, thereby unifying and purifying the text, eliminating confusion and interference, providing a standard and reliable text input for subsequent detection, and improving the accuracy and robustness of prompt word attack detection.

[0036] Then, if the length of the target text after standardization is short, less than the preset length threshold, it can be directly used as the text block to be processed. If the length of the target text after standardization is long, not less than the preset length threshold, the target text can be cut based on semantic cutting technology. The semantic cutting technology is not a brute-force cutting with a fixed window size, but identifies the boundaries where the semantics change and cuts based on the boundaries. The sizes of the text blocks obtained by cutting in the above manner are different, each having a relatively independent semantics.

[0037] In order to ensure the continuity of the context, the present embodiment embeds a context overlap text segment of a configurable length at the beginning and end of each text block cut based on the semantic cutting technology when cutting the target text, to obtain the final text block. This text block with a context overlap text segment ensures that there is an overlap of text content in the boundary area between adjacent text blocks, which can maintain the continuity of the context and prevent the semantics from being fragmented at the block boundary.

[0038] Each text block can also carry metadata, which at least includes the position information of the text block in the original text, which can facilitate subsequent risk positioning and reporting.

[0039] Step S102, perform heuristic semantic analysis on the text block to obtain a first suspicious text block and a first risk score of the first suspicious text block.

[0040] After the text blocks are obtained, the first stage of the prompt word attack detection is entered, and heuristic semantic analysis is performed on the text blocks to obtain a first suspicious text block and a first risk score of the first suspicious text block.

[0041] In this embodiment, at least one of a regular expression library, a keyword dictionary, and information entropy detection can be used to perform heuristic semantic analysis on the text blocks, and a low-level risk score of each text block is determined. If the risk score of a text block is not greater than a preset score threshold (for example, 0, that is, ≤0), the text block is preliminarily determined to be safe; if the risk score of a text block is greater than the preset score threshold (that is, >0), the text block is determined to be a first suspicious text block, and the risk score is the first risk score.

[0042] The first suspicious text block and context information thereof are transmitted to the second stage described in step S103 for in-depth analysis.

[0043] The above embodiment performs high-speed and low-cost heuristic scanning on the text blocks, aims to quickly identify and mark those obvious and known malicious patterns, can quickly determine the first suspicious text block with extremely low overhead, significantly improves the system throughput, focuses the limited computing resources on subsequent more in-depth risk content analysis, and accurately focuses on suspicious text content for subsequent in-depth analysis.

[0044] In step S103, a first language model that is pre-trained is used to perform semantic analysis on the first suspicious text block and context information thereof to obtain a second suspicious text block and a second risk score of the second suspicious text block.

[0045] After the first suspicious text block output by the first stage is obtained, the second stage of the prompt word attack detection is entered, and a first language model that is pre-trained is used to perform semantic analysis on the first suspicious text block and context information thereof to obtain a second suspicious text block and a second risk score of the second suspicious text block. The goal of the second stage is to understand the real intention hidden behind the first suspicious text block to detect complex attacks implemented through language camouflage means.

[0046] In the embodiment, the first language model may be, for example, a model based on a Transformer architecture and trained using prompt samples labeled with classification tags. The first language model may be used as a multi-class classifier to perform semantic analysis on the category of the first suspicious text block according to the first suspicious text block, its location information, and context information, to determine multiple classification tags of each first suspicious text block and the confidence of each classification tag. The classification tags may include: Benign, Instructional Hijacking, Data Exfiltration Attempt, Role-Playing Attack, and the like.

[0047] The first suspicious text block may detect multiple classification tags. In the embodiment, the classification tag with the highest confidence is taken as the target classification tag. If the target classification tag of the first suspicious text block is a non-benign tag, i.e., a classification tag indicating an attack risk such as Instructional Hijacking, Data Exfiltration Attempt, and Role-Playing Attack, the first suspicious text block is determined as a second suspicious text block.

[0048] The second risk score is determined according to the target confidence corresponding to the target classification tag of the second suspicious text block. For example, different confidences correspond to different risk scores, and the second risk score corresponding to the target confidence is determined based on the correspondence between the confidence and the risk score.

[0049] The second suspicious text block and its context information are passed to the third stage described in step S104 for attack intent verification.

[0050] The above embodiment can perform semantic analysis on the first suspicious text block and its context information by using the first language model, can perform deep semantic understanding on the first suspicious text block, can effectively identify hidden and variant attacks, and can greatly reduce false negatives and false positives, thereby improving detection accuracy.

[0051] In step S104, a second language model pre-trained is used to perform structured verification of the attack intent of the second suspicious text block according to preset meta-prompt words, to obtain an attack intent verification result. The attack intent verification result includes a third suspicious text block and a third risk score of the third suspicious text block.

[0052] After obtaining the second suspicious text block in the second stage output, a third stage of the prompt word attack detection is entered, and a pre-trained second language model is used to perform structured verification of the attack intention on the second suspicious text block according to a preset meta prompt word, to obtain an attack intention verification result. The third stage uses a powerful second language model to simulate the reasoning process of a security analyst, and provides a final, expert-level decision for the second suspicious text block identified in the second stage.

[0053] In this embodiment, the second language model can be a large language model, such as a GPT-4 or Claude series model. The second language model is used to perform structured verification of the attack intention on the second suspicious text block according to the preset meta prompt word, guided questions, the second suspicious text block, and context information of the second suspicious text block, to obtain an attack intention verification result. The attack intention verification result includes a third suspicious text block, a third risk score of the third suspicious text block, and a decision reason of the third suspicious text block.

[0054] The above embodiment uses a second language model as a decision maker in combination with a meta prompt word to perform expert-level intention verification on a high-risk second suspicious text block. In this case, under the premise of extremely low false positives, the new, complex, and deeply obfuscated attacks can be accurately identified, and an interpretable attack intention verification result can be output, greatly reducing the false interception and improving the robustness and credibility in an open environment.

[0055] In step S105, a target response action is performed on the original text based on the first risk score, the second risk score, and the third risk score.

[0056] In this embodiment, the suspicious text blocks and risk scores in the three stages of the prompt word attack detection are counted, a configurable weighting algorithm is used to weight and sum the first risk score, the second risk score, and the third risk score, a target risk score for the original text as a whole is determined, and a target response action is performed on the original text according to the target risk score. The target response action can include release, marking, interception, or report generation.

[0057] To better understand the embodiments of the present disclosure, the following describes each step in the prompt word attack detection method of the large language model in more detail.

[0058] For step S101, determining a text block to be processed in the original text, the present embodiment can include: First, the original text is standardized to obtain a target text.

[0059] In specific embodiments, a series of standardization cleaning operations are performed on the original text, including but not limited to: uniform character encoding to Unicode paradigm, removing non-printable control characters, and decoding any detected encoding format (such as URL encoding, Base64 encoding) to restore the original form of the original text. Through the above standardization processing, the target text is obtained.

[0060] The embodiment can effectively eliminate the bypass means achieved by the attacker through character confusion and format transformation, etc., by standardizing the original text into a pure and consistent form, providing a reliable analysis foundation for subsequent detection, and significantly improving the robustness of the anti-escape attack.

[0061] Secondly, if the length of the target text is less than the preset length threshold, the target text is determined as a text block. If the length of the target text is not less than the preset length threshold, the target text is semantically segmented to obtain a text block.

[0062] In the face of long text detection, there are currently problems such as how to efficiently and accurately identify local malicious instructions embedded in a large amount of benign text, and overcome detection failure caused by signal dilution. To solve this problem, in the embodiment, when the length of the target text is not less than the preset length threshold (such as 512 tokens), the core intelligent blocking algorithm will be activated. The intelligent blocking algorithm (i.e., the semantic-based segmentation technique) does not use a fixed window size to perform brute-force segmentation on the target text, but uses a semantic-based segmentation technique to segment the target text into text blocks that are relatively independent in semantics and contain context information, which is convenient for analysis by downstream modules.

[0063] In the embodiment of segmenting the target text to obtain a text block, the following can be included: A preset sentence converter model is used to generate vector embeddings for each sentence in the target text. The cosine distance between the vector embeddings corresponding to each two adjacent sentences is determined. The sentence boundary at which the semantic change occurs is determined according to the cosine distance. The target text is segmented into text blocks based on the sentence boundary. The adjacent text blocks include overlapping text of a first preset length.

[0064] Specifically, a lightweight sentence converter model can be used to generate vector embeddings (Embedding) for each sentence in the target text. The cosine distance between the vector embeddings corresponding to each two adjacent sentences is calculated. It can be understood that the size of the cosine distance can represent the size of the semantic change between the two sentences.

[0065] Based on this, if the cosine distance is greater than the preset distance threshold, it can be determined that there is a relatively obvious semantic change between the two sentences, such as a paragraph or a topic conversion. Therefore, the two sentences can be used as the sentence boundary of semantic segmentation, and the target text is segmented at the sentence boundary.

[0066] In order to ensure the continuity of the context, when the target text is segmented at the sentence boundary, a first preset length of context overlapping text segment is embedded at the head and tail of each segmented text block, such as overlapping one or two sentences, to obtain the final text block. That is, the final generated text blocks will contain an overlapping area of a configurable length.

[0067] Each generated text block is attached with metadata, which at least includes the accurate position information of the text block in the original text, so as to facilitate subsequent risk positioning and reporting.

[0068] In the face of the problem of long text detection failure, the embodiment discards the simple and fixed length segmentation method for long text in the traditional method, and instead adopts a novel intelligent segmentation and context maintenance mechanism. The embodiment first uses the cosine distance to identify the sentence boundary where the semantic change occurs in the target text, such as a paragraph or a topic conversion. On this basis, the target text is segmented into text blocks with overlapping text based on the sentence boundary. In this way, when each text block is analyzed, not only its own content is focused on, but also its overlapping context information is analyzed in depth, thereby effectively preventing malicious instruction signals from being diluted on a macro scale, so that the semantic analysis model can accurately assess the true intention of any given sentence or paragraph in its specific context. This fundamentally solves the "needle in a haystack" problem that plagues single machine learning models.

[0069] For step S102, the text block is subjected to heuristic semantic analysis to obtain a first suspicious text block and a first risk score of the first suspicious text block. The embodiment can determine the first suspicious text block and the first risk score of the first suspicious text block by at least one of the following implementation modes.

[0070] Implementation mode one. A preset regular expression library is used to perform semantic matching on the text block to obtain the first suspicious text block; wherein the regular expression library includes multiple types of patterns related to prompt word injection.

[0071] Specifically, a highly optimized regular expression library (for example, Google's RE2 library) can be used to perform semantic matching on the text block. The regular expression library can include a large number of patterns related to prompt word injection, such as instructional phrases, API key formats, and other structural features of sensitive data.

[0072] Among them, the imperative phrase is used to match common malicious instruction templates such as "ignore instructions", "print your prompt". The structural features of other sensitive data are used to identify sensitive information with specific formats such as credit card numbers, ID numbers, etc.

[0073] The text block to be detected is scanned and matched with all patterns in the regular expression library. When the text block is successfully matched with any pattern, a rule hit is triggered, and the text block is marked as the first suspicious text block, and the first suspicious text block is assigned a first risk score.

[0074] The first implementation can realize high-performance and low-cost preliminary screening for known attack patterns.

[0075] And / or, implementation two. According to the preset keyword dictionary, the text block is semantically matched to obtain the first suspicious text block; wherein the keyword dictionary includes keywords related to attack behavior.

[0076] The embodiment can construct and maintain a continuously updated keyword dictionary, and quickly semantically match the text block to identify potential attack intentions. The keyword dictionary contains keywords and phrases highly related to prompt word injection, jailbreaking, data theft and other attack behaviors.

[0077] In specific implementation, an efficient string matching algorithm (such as Aho-Crashick automatic machine algorithm) can be used to compare the text block with all keywords in the keyword dictionary. When the number of successfully matched keywords or the weighted risk value in the text block exceeds the preset threshold, the text block is marked as the first suspicious text block, and the first suspicious text block is assigned a first risk score.

[0078] The second implementation quickly matches the text block based on the keyword dictionary, which can accurately capture the first suspicious text block of known attack patterns with extremely low overhead, and realize high-performance preliminary threat screening.

[0079] And / or, implementation three. Calculate the information entropy of the text block, and determine the text block with information entropy higher than the preset information entropy as the first suspicious text block.

[0080] Specifically, the information entropy of the text block is calculated, which is specifically, for example, Shannon entropy. Since the malicious payload after encoding or encryption usually shows higher information entropy than natural language, the information entropy tag can effectively detect the malicious code after obfuscation. Further, the embodiment determines the text block with information entropy higher than the preset information entropy as the first suspicious text block, and assigns a first risk score to the first suspicious text block.

[0081] The first suspicious text block is determined based on the above implementation manner, and then a first risk score is determined for the first suspicious text block.

[0082] The embodiment assigns a low-level risk score to each first suspicious text block. If the risk score of a text block is not greater than 0, it can be preliminarily determined to be safe. If the risk score is greater than 0, the text block is a first suspicious text block that needs to be analyzed in depth, and a first risk score is determined for the first suspicious text block.

[0083] The embodiment can quickly identify and mark obvious and known first suspicious text blocks through regular expression libraries, keyword dictionaries, and information entropy, significantly improve system throughput, focus limited computing resources on subsequent more in-depth risk content analysis, and accurately focus suspicious text content for subsequent in-depth analysis.

[0084] For step S103, the first suspicious text block and its context information are subjected to semantic analysis by using a pre-trained first language model to obtain a second suspicious text block and a second risk score of the second suspicious text block. The embodiment can include the following steps (1)-(4).

[0085] (1) Obtain a pre-trained first language model; wherein the first language model is a model based on a Transformer architecture and is trained by using prompt samples labeled with classification labels.

[0086] Specifically, the first language model can be a language model based on a Transformer architecture and fine-tuned. In order to balance performance and accuracy, the embodiment can select a distilled version of a model optimized for classification tasks (such as DistilledBERT, RoBERTa) or a model designed specifically for understanding tasks (such as DeBERTa).

[0087] The first language model is obtained by supervised training using a large number of prompt samples labeled with classification labels; wherein the classification labels include benign, instruction hijacking, data theft attempt, role-playing attack, etc.

[0088] Based on the above prompt samples, the first language model is trained as a multi-class classifier, which can label each input text block with a fine classification label.

[0089] (2) Input the first suspicious text block and its context information into the first language model; wherein the context information of the first suspicious text block includes: position information of the first suspicious text block in the original text, and overlapping text of a second preset length between the first suspicious text block and adjacent text blocks.

[0090] To make a more accurate judgment, the input of the first language model includes not only the current first suspicious text block to be analyzed, but also metadata such as its position information, and also includes second preset length of overlapping text from adjacent text blocks.

[0091] (3) According to the first suspicious text block and its context information, the first language model determines the target classification label corresponding to the first suspicious text block and a second risk score of the target classification label.

[0092] The first language model determines the confidence of each classification label corresponding to the first suspicious text block according to the first suspicious text block and its context information. The classification label with the highest confidence is determined as the target classification label, and the highest confidence can be quantified as a second risk score. This second risk score can not only reflect the confidence of the current classification result of the first language model, but also provide a fine-grained risk assessment basis for downstream decision and response.

[0093] (4) The first suspicious text block with a second risk score higher than a preset score value is determined as a second suspicious text block. The second suspicious text block is sent to the third stage for verification of attack intent.

[0094] The first language model introduced in this embodiment can improve the detection accuracy of complex, hidden and semantic prompt word injection attacks by combining context information. First, the first language model not only analyzes the first suspicious text block itself, but also integrates its position information in the original text and overlapping text with adjacent blocks, effectively reconstructing the local context. This enables the first language model to understand the malicious instruction logic across sentences and paragraphs, and accurately identify advanced attacks that are diluted or disguised in long text.

[0095] Secondly, by outputting specific target classification labels and second risk scores, this embodiment not only makes a binary judgment, but also reveals the nature and intent of the attack, providing more valuable decision basis for security audit and subsequent response.

[0096] In addition, this embodiment only performs in-depth analysis on the first suspicious text block in the first stage, and the amount of data processed is small, avoiding the performance overhead of sending all original text into a complex model, thereby greatly increasing the detection depth and breadth without significantly increasing system latency.

[0097] Current existing prompt word injection detection schemes mostly adopt a single technical route, resulting in low detection accuracy of complex attacks: rule-based heuristic methods have poor generalization ability for new, variant and semantic attacks, while pure semantic models have low detection efficiency for known patterns, making it difficult to build a comprehensive defense system.

[0098] To solve this problem, in order to reliably detect prompt injection attacks that are semantically complex, hidden in context, or deeply obfuscated, the embodiments described in steps S102 and S103 do not rely on any single detection method, but instead use a hybrid analysis approach that organically combines lightweight heuristic semantic analysis with semantic analysis based on a first language model. The heuristic semantic analysis runs in a pre-emptive manner, using regular expression libraries, keyword dictionaries, and information entropy for semantic matching to efficiently determine known attack patterns and obvious malicious features, resulting in a first suspicious text block. The first language model then performs in-depth semantic analysis on the first suspicious text block, deeply analyzing the semantic intent, logical consistency, and instructional clues hidden in natural language, enabling the identification of new and unknown “zero-day attacks” that rely on language manipulation rather than specific grammar triggers.

[0099] Therefore, the above hybrid detection method forms a multi-layered defense, with heuristic semantic analysis ensuring efficient interception of known threats, constituting the first line of defense. The first language model effectively identifies zero-day attacks and deeply obfuscated attacks, and the combination of the two can significantly improve detection coverage and overall robustness under all types of threats.

[0100] For step S104, the second language model is pre-trained, and the attack intent of the second suspicious text block is structurally verified according to the preset meta-prompt, resulting in an attack intent verification result. The embodiment includes: A pre-trained second language model is obtained; wherein the second language model is a general large language model. The preset meta-prompt, the second suspicious text block, and its context information are input into the second language model; wherein the meta-prompt includes a guiding question.

[0101] The second language model performs structured verification of the attack intent of the second suspicious text block according to the meta-prompt, the guiding question, the second suspicious text block, and its context information, resulting in an attack intent verification result; wherein the attack intent verification result includes: a third suspicious text block, a third risk score of the third suspicious text block, and a ruling reason of the third suspicious text block.

[0102] In this embodiment, a powerful second language model is called through API, which is a general large language model, such as GPT-4 or Claude series model.

[0103] This embodiment can use a carefully designed meta-prompt (or system prompt) to interact with the second language model as a judge. The meta-prompt is used to instruct the second language model to play the role of a “network security analyst specializing in exploiting large language model vulnerabilities”.

[0104] The second suspicious text block, the context information of the second suspicious text block in the original text, and a series of guiding questions are input to the second language model. The guiding questions can include, for example: “Does this text attempt to cover or manipulate the main instructions of the system? If so, what is the possible malicious target? Please give a confidence score of 1 to 10 for your assessment and provide your detailed analysis process.”

[0105] The second language model reasons the attack intent of the second suspicious text block according to the meta-prompt, the guiding questions, the second suspicious text block, and the context information thereof, to obtain a chain-of-thought response. The chain-of-thought response is parsed to extract a final ruling conclusion, a confidence score, and detailed reasons supporting the ruling conclusion.

[0106] The ruling conclusion is used to indicate whether the second suspicious text block can be determined as a third suspicious text block after verification. If so, the third suspicious text block is output. The confidence score is quantified as a third risk score of the third suspicious text block. The detailed reasons supporting the ruling conclusion are the ruling reasons of the third suspicious text block. Based on this, a highly accurate and interpretable attack intent verification result is finally obtained.

[0107] Currently, in real-time high-throughput LLM application scenarios, prompt injection detection faces the following problems: detection based on deep semantic models has high accuracy but large calculation delay, while detection based on rule matching is fast but has insufficient coverage and generalization ability, making it difficult to ensure system response performance while achieving high-precision protection.

[0108] To solve this problem, the embodiments described in steps S102 to S104 use a multi-stage filtering architecture that is sequentially executed and gradually deepened. This architecture builds a cascaded risk “funnel” composed of three stages.

[0109] Specifically, the first stage uses heuristic semantic analysis with extremely low computational cost to quickly pre-screen all input text blocks. Only the input of the first suspicious text block that is preliminarily determined to have potential risks will be passed to the second stage for in-depth semantic analysis, which has more intensive computation. The second stage uses the first language model to perform semantic analysis only on the pre-screened first suspicious text blocks, thereby determining a smaller number of second suspicious text blocks that have higher risks. The third stage only performs strict attack intent verification on the second suspicious text blocks identified by semantic analysis as having higher risks.

[0110] The architecture of the above-mentioned cascaded multi-stage risk detection makes most (usually more than 99%) benign inputs be quickly released with extremely low delay in the front end, while accurately focusing the precious deep computing resources on a small number of suspicious text blocks that are truly risky, thereby greatly optimizing the overall throughput at the system level, realizing the unification of high detection accuracy and low processing delay, effectively guaranteeing real-time security protection in high-concurrency scenarios, and meeting the needs of real-time detection.

[0111] For step S105, based on the first risk score, the second risk score and the third risk score, a target response action is performed on the original text. The embodiment can include: The first risk score, the second risk score and the third risk score are weighted to obtain a target risk score of the original text; according to the corresponding relationship between the risk score and the response action, a target response action corresponding to the target risk score is determined; and the target response action is performed on the original text.

[0112] The embodiment collects and integrates the risk scores and suspicious text blocks from all previous stages, calculates the final risk score for the original text as a whole, and determines which response action to take accordingly.

[0113] In an implementation manner, a configurable weighting algorithm can be used to fuse the first risk score, the second risk score and the third risk score. For example, the weight of the first risk score from the first stage is lower than the weight of the second risk score from the second stage, and the weight of the second risk score from the second stage is lower than the weight of the third risk score from the third stage. The above weight configuration indicates that the more suspicious text blocks confirmed by subsequent and more complex analysis stages, the higher the confidence and value of the detection result. Therefore, when the first risk score, the second risk score and the third risk score are weighted, higher weights are assigned to the risk scores of subsequent stages, which in essence allows suspicious text blocks with higher confidence and greater decision value to dominate in the final decision, so that the final target risk score is more accurate and reliable.

[0114] Further, the first risk score, the second risk score and the third risk score are weighted to obtain the final target risk score of the original text. And according to the corresponding relationship between the risk score and the response action, a target response action corresponding to the target risk score is determined. The corresponding relationship between the risk score and the response action can refer to the following Table 1 example: Table 1: Corresponding relationship between risk score and response action

[0115] According to the correspondence between the above risk score and the response action, when the target risk score is lower than the threshold value $T_1$, the original text is determined to be safe, and the release can be performed on it, and it is passed to the target LLM.

[0116] When the target risk score is between the threshold values $T_1$ and $T_2$ ($T_1 \le \text{Score} < T_2$), the original text is allowed to be passed to the target LLM, but at the same time a flag is executed to generate a safe warning log for manual review.

[0117] When the target risk score is higher than or equal to the threshold value $T_2$, the original text is intercepted, and an error or warning information is returned to the user. Alternatively, the present embodiment can also perform a disinfection operation on the original text, which can be understood as: removing the detected third suspicious text block with maliciousness, or removing the first suspicious text block, the second suspicious text block and the third suspicious text block. After that, the remaining text after the disinfection operation is passed to the target LLM.

[0118] On the basis of the above embodiment, for any original text that is flagged or intercepted, a detailed JSON format report can be generated, which contains the final decision (i.e. the first suspicious text block, the second suspicious text block and the third suspicious text block), the target risk score, and the detailed findings from each analysis stage.

[0119] Referring to Figure 2 In an actual application scenario, the prompt word attack detection method of the large language model provided by the present disclosure can be applied to an end-to-end workflow, as shown below: Start: When a user submits an original text to a target large language model through an application program of a user terminal, the user terminal initiates a text detection request carrying the original text to the target large language model.

[0120] Request interception: in response to the text detection request of the user terminal, the text detection request is intercepted, and the original text to be subjected to prompt word attack detection is obtained. Specifically, the text detection request is intercepted as a middleware to obtain the original text carried by the text detection request.

[0121] Preprocessing and blocking: standardizing the original text to obtain a target text; if the length of the target text is not less than a preset length threshold, the target text is subjected to semantic segmentation to obtain a text block containing overlapping text.

[0122] First-stage scanning: heuristic semantic parsing of the text blocks, obtaining first suspicious text blocks and first risk scores of the first suspicious text blocks. Specifically, all text blocks are sent to the fast heuristic scanning stage, most of the benign content is quickly identified and cleared in this stage, and the first risk score of the benign content is not greater than 0; and the first suspicious text blocks with a first risk score greater than 0 are passed to the next stage.

[0123] Second-stage analysis: semantic analysis of the first suspicious text blocks and their context information by a pre-trained first language model, obtaining second suspicious text blocks and second risk scores of the second suspicious text blocks. Specifically, the first suspicious text blocks from the first stage are analyzed in depth by the context-aware semantic analysis first language model, obtaining the second risk score of the first suspicious text blocks in this stage. If the second risk score is lower than the preset score value, it is determined as low risk, and the low-risk first suspicious text blocks are cleared; if the second risk score is not lower than the preset score value, it is determined as high risk, and the high-risk first suspicious text blocks are passed to the next stage as second suspicious text blocks.

[0124] Third-stage verification: structured verification of the attack intent of the second suspicious text blocks by a pre-trained second language model according to the preset meta-prompt, obtaining an attack intent verification result. Specifically, the second suspicious text blocks with the highest risk from the second stage are sent to the third stage for attack intent verification, and a powerful second language model is used for final and interpretable decision.

[0125] Comprehensive decision and action response: based on the first risk score, the second risk score and the third risk score, performing a target response action on the original text. Specifically, all suspicious text blocks and risk scores from the entire processing pipeline are collected, and the final target risk score for the original text is calculated by a weighting algorithm. Based on the target risk score, a corresponding target response action is performed on the original text, such as release, marking or interception, and a corresponding response message and detailed analysis report are generated.

[0126] End: the processing flow ends. By this time, the large language model in the back end has been successfully protected from potential attacks, and a benign request has been safely delivered.

[0127] In order to more clearly show the difference between the prompt word attack detection method of the large language model provided by the present disclosure and the related art, the key performance indicators of different technical solutions are compared in Table 2 as follows: Table 2: Technical comparison

[0128] In summary, the method for detecting prompt word attacks of a large language model provided by the embodiment of the present application comprises: determining a text block to be processed in an original text; performing heuristic semantic analysis on the text block to obtain a first suspicious text block and a first risk score of the first suspicious text block; performing semantic analysis on the first suspicious text block and context information thereof by using a first pre-trained language model to obtain a second suspicious text block and a second risk score of the second suspicious text block; performing structured verification of an attack intention on the second suspicious text block according to a preset meta prompt word by using a second pre-trained language model to obtain an attack intention verification result; wherein the attack intention verification result comprises a third suspicious text block and a third risk score of the third suspicious text block; and performing a target response action on the original text based on the first risk score, the second risk score and the third risk score.

[0129] The technical solution can realize full-process coverage from rapid preliminary screening to deep semantic analysis and attack intention verification by constructing a three-level progressive detection method. First, heuristic semantic analysis is used as a preliminary screening link, which can complete the preliminary filtering of large-scale text blocks at very low cost, effectively guarantee the system basic throughput performance, greatly reduce the calculation cost of subsequent deep analysis, and guarantee the processing efficiency. Second, the first language model performs context-aware semantic analysis on the first suspicious text block, which can identify medium-complexity attacks with language camouflage, accurately select high-risk second suspicious text blocks, and improve the detection accuracy. Finally, the second language model is used to perform structured verification of the attack intention in combination with the meta prompt word, which ensures that no complex hidden attacks are missed, realizes accurate identification of new and complex attack intentions, and makes explainable decisions. The above multi-stage cooperative detection scheme forms a defense-in-depth system, which not only maintains the real-time performance of the system through front-end rapid filtering, but also ensures the detection accuracy through back-end deep analysis, avoids single-dimensional misjudgment, and makes the target response action more scientific. Therefore, the present solution can not only meet the efficient processing needs of long texts, but also improve the detection accuracy of complex attacks, providing reliable protection for the safe application of large language models.

[0130] Corresponding to the above method embodiment, the embodiment of the present application provides a prompt word attack detection system for a large language model, as shown in Figure 3 The system comprises the following modules: a text block determination module, a first semantic analysis module, a second semantic analysis module, an attack intention verification module and an action response module. Wherein: The text block determination module 301 is configured to determine a text block to be processed in an original text; wherein the text block is marked with position information; The first semantic analysis module 302 is configured to perform heuristic semantic analysis on the text block to obtain a first suspicious text block and a first risk score of the first suspicious text block; The second semantic analysis module 303 is configured to perform semantic analysis on the first suspicious text block and context information thereof by using a pre-trained first language model, to obtain a second suspicious text block and a second risk score of the second suspicious text block. The attack intention verification module 304 is configured to perform structured verification of an attack intention of the second suspicious text block according to a preset meta-prompt word by using a pre-trained second language model, to obtain an attack intention verification result. The attack intention verification result includes a third suspicious text block and a third risk score of the third suspicious text block. The action response module 305 is configured to perform a target response action on the original text based on the first risk score, the second risk score, and the third risk score.

[0131] The prompt word attack detection system for large language models provided in this embodiment is implemented by using a modular and pipeline architecture, so as to ensure high cohesion and low coupling characteristics of the system, and facilitate maintenance and extension. The system architecture design enables data to sequentially flow through each detection stage, each module performs a highly specialized analysis task, and the result is passed to the next module.

[0132] The prompt word attack detection system for large language models is composed of five core modules: the text block determination module 301, the first semantic analysis module 302 in the first stage, the second semantic analysis module 303 in the second stage, the attack intention verification module 304 in the third stage, and the action response module 305. The system can be deployed as a security middleware or an API (Application Programming Interface) at an endpoint, and is used to intercept and analyze all input requests flowing to a target large language model.

[0133] In some embodiments, the text block determination module 301 further includes: A standardization unit configured to perform standardization processing on the original text to obtain a target text. A first text block determination unit configured to determine the target text as a text block if a length of the target text is less than a preset length threshold. A second text block determination unit configured to perform semantic segmentation on the target text to obtain the text block if the length of the target text is not less than the preset length threshold.

[0134] In some embodiments, the second text block determination unit is further configured to: Generate a vector embedding for each sentence in the target text by using a preset sentence converter model. Determine a cosine distance between the vector embeddings corresponding to each two adjacent sentences. determine a semantic change boundary according to the cosine distance; split the target text into text blocks based on the semantic change boundary; wherein, the overlapping text of a first preset length is included between adjacent text blocks.

[0135] In some embodiments, the first semantic analysis module 302 is further configured to: perform semantic matching on the text blocks using a preset regular expression library to obtain first suspicious text blocks; wherein, the regular expression library includes multiple types of patterns related to prompt word injection; and / or, perform semantic matching on the text blocks according to a preset keyword dictionary to obtain the first suspicious text blocks; wherein, the keyword dictionary includes keywords related to attack behavior; and / or, calculate the information entropy of the text blocks, and determine the text blocks with information entropy higher than a preset information entropy as the first suspicious text blocks; determine a first risk score for the first suspicious text blocks.

[0136] In some embodiments, the second semantic analysis module 303 is further configured to: obtain a pre-trained first language model; wherein, the first language model is a model based on a Transformer architecture and is trained using prompt word samples labeled with classification labels; input the first suspicious text blocks and their context information into the first language model; wherein, the context information of the first suspicious text blocks includes: the position information of the first suspicious text blocks in the original text, and the overlapping text of a second preset length between the first suspicious text blocks and adjacent text blocks; determine, by the first language model, the target classification label corresponding to the first suspicious text blocks and a second risk score of the target classification label according to the first suspicious text blocks and their context information; determine the first suspicious text blocks with the second risk score higher than a preset score value as second suspicious text blocks.

[0137] In some embodiments, the attack intent verification module 304 is further configured to: obtain a pre-trained second language model; wherein, the second language model is a general large language model; input a preset meta prompt word, the second suspicious text blocks and their context information into the second language model; wherein, the meta prompt word includes a leading question; The second language model is used to perform structured verification of attack intention on the second suspicious text block according to the meta-prompt word, the guiding question, the second suspicious text block and context information thereof, to obtain an attack intention verification result; wherein the attack intention verification result includes a third suspicious text block, a third risk score of the third suspicious text block and a ruling reason of the third suspicious text block.

[0138] In some embodiments, the action response module 305 is further configured to: perform weighted scoring on the first risk score, the second risk score and the third risk score to obtain a target risk score of the original text; determine a target response action corresponding to the target risk score according to a corresponding relationship between risk scores and response actions; perform the target response action on the original text.

[0139] In some embodiments, the prompt word attack detection system of the large language model further comprises: An interception module is configured to, in response to a text detection request of a user end, intercept the text detection request and obtain an original text to be subjected to prompt word attack detection.

[0140] The prompt word attack detection system of the large language model provided in this embodiment has the same technical features as the prompt word attack detection method of the large language model provided in the above embodiments, and can also solve the same technical problems and achieve the same technical effects. For brevity, the part of the embodiments not mentioned in the foregoing description can be referred to the corresponding content in the foregoing prompt word attack detection method of the large language model.

[0141] This embodiment further provides an electronic device, a structural schematic diagram of which is shown in Figure 4 The device includes a processor 101 and a memory 102; wherein the memory 102 is configured to store one or more computer instructions, and the one or more computer instructions are executed by the processor to implement the prompt word attack detection method of the large language model.

[0142] Figure 4 The electronic device shown in the figure further includes a bus 103 and a communication interface 104, and the processor 101, the communication interface 104 and the memory 102 are connected through the bus 103.

[0143] The memory 102 can include a high-speed random access memory (RAM) and can also include a non-volatile memory, such as at least one disk memory. The bus 103 can be an ISA bus, a PCI bus, or an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one bidirectional arrow is used in the figure to represent only one bus or one type of bus.

[0144] The communication interface 104 is used to connect with at least one user terminal and other network elements through a network interface, and to send the encapsulated IPv4 packet or IPv4 packet to the user terminal through the network interface.

[0145] The processor 101 can be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above method can be completed by integrated logic circuits of hardware in the processor 101 or instructions in the form of software. The above processor 101 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. Each method, step and logic block disclosed in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present disclosure can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, or other mature storage media in the art. The storage medium is located in the memory 102, and the processor 101 reads the information in the memory 102 and combines the hardware to complete the steps of the method of the above embodiments.

[0146] The embodiment of the present application also provides a readable storage medium, and the readable storage medium stores a computer program. When the computer program is run by a processor, the steps of the prompt word attack detection method of the large language model of the foregoing embodiment are executed.

[0147] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. The above described apparatus embodiments are merely exemplary. For example, the division of the units is only a logical function division, and there can be another division manner in actual implementation; for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different units, can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0148] The units described as separated components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. In actual implementation, some or all of the units can be selected according to the actual needs to achieve the purposes of the embodiments of the present application.

[0149] In addition, each functional unit in the various embodiments of the present application can be integrated in one processing unit, or each unit can exist physically as a separate unit, or two or more units can be integrated in one unit.

[0150] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer readable storage medium of a processor executable. Based on such understanding, the technical solutions of the present application or the part of the present application that makes essential contributions to the prior art or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0151] Finally, it should be noted that the above-described embodiments are merely specific embodiments of the present application, which are used to illustrate the technical solutions of the present application, but not to limit the same. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those skilled in the art that any person skilled in the art can still modify or easily think of changes to the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some of the technical features, within the technical scope disclosed by the present application. The modifications, changes or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A prompt word attack detection method of a large language model, characterized in that, The method comprises the following steps: determining a text block to be processed in the original text; performing heuristic semantic analysis on the text block to obtain a first suspicious text block and a first risk score of the first suspicious text block; performing semantic analysis on the first suspicious text block and its context information by using a pre-trained first language model to obtain a second suspicious text block and a second risk score of the second suspicious text block; performing structured verification of attack intent on the second suspicious text block according to a preset meta-prompt word by using a pre-trained second language model to obtain an attack intent verification result; wherein the attack intent verification result comprises a third suspicious text block and a third risk score of the third suspicious text block; performing a target response action on the original text based on the first risk score, the second risk score and the third risk score.

2. The prompt attack detection method of a large language model according to claim 1, characterized in that, The determination of the text block to be processed in the original text comprises: performing standardization processing on the original text to obtain a target text; if the length of the target text is less than a preset length threshold, determining the target text as a text block; if the length of the target text is not less than the preset length threshold, performing semantic segmentation on the target text to obtain the text block.

3. The prompt attack detection method of a large language model according to claim 2, characterized in that, The semantic segmentation of the target text to obtain the text block comprises: generating vector embeddings for each sentence in the target text by using a preset sentence converter model; determining the cosine distance between the vector embeddings corresponding to each two adjacent sentences; determining the sentence boundary at which a semantic change occurs according to the cosine distance; segmenting the target text into text blocks based on the sentence boundary; wherein adjacent text blocks comprise overlapping text of a first preset length.

4. The prompt attack detection method of a large language model according to claim 1, characterized in that, The heuristic semantic analysis of the text block to obtain a first suspicious text block and a first risk score of the first suspicious text block comprises: performing semantic matching on the text block by using a preset regular expression library to obtain a first suspicious text block; wherein the regular expression library comprises multiple types of patterns related to prompt word injection; and / or, performing semantic matching on the text block according to a preset keyword dictionary to obtain the first suspicious text block; wherein the keyword dictionary comprises keywords related to attack behavior; and / or, calculating the information entropy of the text block, and determining a text block with an information entropy higher than a preset information entropy as the first suspicious text block; determining a first risk score for the first suspicious text block.

5. The prompt attack detection method of a large language model according to claim 1, characterized in that, The semantic analysis of the first suspicious text block and its context information by using a pre-trained first language model to obtain a second suspicious text block and a second risk score of the second suspicious text block comprises: obtaining a pre-trained first language model; wherein the first language model is a model based on a Transformer architecture and is trained by using prompt word samples labeled with classification labels; inputting the first suspicious text block and context information thereof into the first language model; wherein the context information of the first suspicious text block comprises: position information of the first suspicious text block in the original text, and overlapping text of a second preset length between the first suspicious text block and adjacent text blocks; determining, by the first language model, a target classification label corresponding to the first suspicious text block and a second risk score of the target classification label according to the first suspicious text block and context information thereof; determining, as a second suspicious text block, the first suspicious text block with a second risk score higher than a preset score value.

6. The prompt attack detection method of a large language model according to claim 1, wherein, performing, by the second language model pre-trained, structural verification of attack intent on the second suspicious text block and context information thereof according to a preset meta prompt word, to obtain an attack intent verification result, comprising: obtaining a second language model pre-trained; wherein the second language model is a general large language model; inputting a preset meta prompt word, the second suspicious text block and context information thereof into the second language model; wherein the meta prompt word comprises a guiding question; performing, by the second language model, structural verification of attack intent on the second suspicious text block according to the meta prompt word, the guiding question, the second suspicious text block and context information thereof, to obtain an attack intent verification result; wherein the attack intent verification result comprises: a third suspicious text block, a third risk score of the third suspicious text block, and a ruling reason of the third suspicious text block.

7. The prompt attack detection method of a large language model according to claim 1, characterized in that, performing, based on the first risk score, the second risk score and the third risk score, a target response action on the original text, comprising: performing weighted scoring on the first risk score, the second risk score and the third risk score to obtain a target risk score of the original text; determining a target response action corresponding to the target risk score according to a corresponding relationship between risk scores and response actions; performing the target response action on the original text.

8. The prompt attack detection method of a large language model according to claim 1, wherein, Before determining the text block to be processed in the original text, the method further comprises: in response to a text detection request of a user terminal, intercepting the text detection request and obtaining an original text to be subjected to prompt word attack detection.

9. A prompt word attack detection system of a large language model, characterized by, The system comprises the following modules: a text block determination module for determining a text block to be processed in an original text; wherein the text block is marked with position information; a first semantic analysis module for performing heuristic semantic analysis on the text block to obtain a first suspicious text block and a first risk score of the first suspicious text block; a second semantic analysis module for performing semantic analysis on the first suspicious text block and context information thereof by a first language model pre-trained to obtain a second suspicious text block and a second risk score of the second suspicious text block; The attack intention verification module is configured to perform structured verification of attack intention on the second suspicious text block according to preset meta-prompt words by using a pre-trained second language model, to obtain an attack intention verification result; wherein the attack intention verification result includes a third suspicious text block and a third risk score of the third suspicious text block. The action response module is configured to perform a target response action on the original text based on the first risk score, the second risk score and the third risk score.

10. An electronic device, comprising: The method comprises the following steps: A processor and a storage device; the storage device stores a computer program, and the computer program realizes the steps of the prompt word attack detection method of the large language model according to any one of claims 1 to 8 when the processor runs.

Images