Methods, systems, and electronic devices for detecting cue word attacks in large language models

Abstract

This invention provides a method, system, and electronic device for detecting cue word attacks using a large language model, relating to the field of text processing technology. The method includes: identifying text blocks to be processed in the original text; performing heuristic semantic parsing on the text blocks to obtain a first suspicious text block and its first risk score; performing semantic analysis on the first suspicious text block and its contextual information using a first language model to obtain a second suspicious text block and its second risk score; performing structured verification of the attack intent of the second suspicious text block based on preset meta-cue words using a second language model to obtain an attack intent verification result; the attack intent verification result includes a third suspicious text block and its third risk score; and executing a target response action on the original text based on the first, second, and third risk scores. This invention can both meet the needs of efficient processing of long texts and improve the detection accuracy of complex attacks.

Description

Technical Field

[0001] This invention relates to the field of text processing technology, and in particular to a method, system, and electronic device for detecting cue word attacks in a large language model. Background Technology

[0002] With the widespread application of Large Language Models (LLMs) in fields such as intelligent customer service and content generation, prompt injection attacks have become a serious security threat. Attackers hijack the intended function of the large language model by embedding malicious instructions into the input text, thereby performing unauthorized operations, which poses a serious security problem.

[0003] To combat keyword injection attacks, some techniques employ rule-based and keyword-based filtering methods. However, these methods are highly vulnerable, easily bypassed by attackers through simple synonym substitution, character encoding, or text obfuscation, resulting in extremely high false negative rates. Other techniques utilize single-stage machine learning models. These methods introduce significant computational latency and are prone to failure in long texts due to the dilution of malicious signals, also leading to very high false negative rates.

[0004] Therefore, current detection solutions for prompt injection attacks generally suffer from low detection accuracy and insufficient ability to handle long texts. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a method, system and electronic device for detecting cue word attacks in large language models. This method can both meet the needs of efficient processing of long texts and improve the detection accuracy of complex attacks.

[0006] In a first aspect, embodiments of the present invention provide a method for detecting cue word attacks in a large language model, the method comprising the following steps:

[0007] Identify the text blocks to be processed within the original text;

[0008] Heuristic semantic parsing is performed on the text block to obtain a first suspicious text block and a first risk score for the first suspicious text block;

[0009] Using a pre-trained first language model, semantic analysis is performed on the first suspicious text block and its context information to obtain the second suspicious text block and the second risk score of the second suspicious text block;

[0010] Using a pre-trained second language model, the attack intent of the second suspicious text block is structurally verified based on preset meta-prompt words to obtain the attack intent verification result; wherein, the attack intent verification result includes: a third suspicious text block and a third risk score of the third suspicious text block;

[0011] Based on the first risk score, the second risk score, and the third risk score, a target response action is performed on the original text.

[0012] In some implementations, determining the text block to be processed in the original text includes:

[0013] The original text is standardized to obtain the target text;

[0014] If the length of the target text is less than a preset length threshold, then the target text is determined as a text block;

[0015] If the length of the target text is not less than a preset length threshold, then the target text is semantically segmented to obtain the text block.

[0016] In some implementations, the semantic segmentation of the target text to obtain the text blocks includes:

[0017] A vector embedding is generated for each sentence in the target text using a pre-defined sentence converter model;

[0018] Determine the cosine distance between the vector embeddings corresponding to every two adjacent sentences;

[0019] The semantic boundary of the statement undergoing semantic change is determined based on the cosine distance;

[0020] The target text is divided into text blocks based on the statement boundaries; wherein, adjacent text blocks include overlapping text of a first preset length.

[0021] In some implementations, the heuristic semantic parsing of the text block to obtain a first suspicious text block and a first risk score for the first suspicious text block includes:

[0022] The text block is semantically matched using a pre-defined regular expression library to obtain the first suspicious text block; wherein, the regular expression library includes multiple patterns related to prompt word injection;

[0023] And / or, semantic matching of the text block is performed according to a preset keyword dictionary to obtain the first suspicious text block; wherein, the keyword dictionary includes keywords related to the attack behavior;

[0024] And / or, calculate the information entropy of the text block, and identify the text block whose information entropy is higher than a preset information entropy as the first suspicious text block;

[0025] A first risk score is determined for the first suspicious text block.

[0026] In some implementations, the step of performing semantic analysis on the first suspicious text block and its context information using a pre-trained first language model to obtain a second suspicious text block and a second risk score for the second suspicious text block includes:

[0027] Obtain a pre-trained first language model; wherein the first language model is a model based on the Transformer architecture and is trained using prompt word samples labeled with classification tags;

[0028] The first suspicious text block and its context information are input into the first language model; wherein, the context information of the first suspicious text block includes: the position information of the first suspicious text block in the original text, and the overlapping text of the first suspicious text block and adjacent text blocks of a second preset length;

[0029] The first language model determines the target classification label corresponding to the first suspicious text block and the second risk score of the target classification label based on the first suspicious text block and its context information.

[0030] The first suspicious text block whose second risk score is higher than the preset score value is identified as the second suspicious text block.

[0031] In some implementations, the step of using a pre-trained second language model to perform structured verification of the attack intent of the second suspicious text block and its context information based on preset meta-cue words, and obtaining the attack intent verification result, includes:

[0032] Obtain a pre-trained second language model; wherein, the second language model is a general-purpose large language model;

[0033] The preset meta-prompt words, the second suspicious text block, and their contextual information are input into the second language model; wherein, the meta-prompt words include guiding questions;

[0034] The second language model performs a structured verification of the attack intent of the second suspicious text block based on the meta-prompt words, the guiding questions, the second suspicious text block and its contextual information, to obtain an attack intent verification result; wherein, the attack intent verification result includes: a third suspicious text block, a third risk score of the third suspicious text block and a ruling reason for the third suspicious text block.

[0035] In some implementations, performing a target response action on the original text based on the first risk score, the second risk score, and the third risk score includes:

[0036] The first risk score, the second risk score, and the third risk score are weighted and scored to obtain the target risk score of the original text.

[0037] Based on the correspondence between risk scores and response actions, determine the target response action corresponding to the target risk score;

[0038] Perform the target response action on the original text.

[0039] In some implementations, before determining the text block to be processed in the original text, the method further includes:

[0040] In response to a text detection request from the user, the text detection request is intercepted, and the original text to be detected for prompt word attacks is obtained.

[0041] Secondly, embodiments of the present invention provide a cue word attack detection system for a large language model, the system comprising the following modules:

[0042] A text block determination module is used to determine the text blocks to be processed in the original text; wherein, the text blocks are marked with position information;

[0043] The first semantic analysis module is used to perform heuristic semantic analysis on the text block to obtain a first suspicious text block and a first risk score for the first suspicious text block;

[0044] The second semantic analysis module is used to perform semantic analysis on the first suspicious text block and its context information using a pre-trained first language model, so as to obtain the second suspicious text block and the second risk score of the second suspicious text block;

[0045] The attack intent verification module is used to perform structured verification of the attack intent of the second suspicious text block based on preset meta-prompt words using a pre-trained second language model, and obtain the attack intent verification result; wherein, the attack intent verification result includes: a third suspicious text block and a third risk score of the third suspicious text block;

[0046] An action response module is used to perform a target response action on the original text based on the first risk score, the second risk score, and the third risk score.

[0047] Thirdly, the embodiments of the invention also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, wherein when the processor executes the computer program, it implements the steps of the prompt word attack detection method for the large language model mentioned in the first aspect.

[0048] The embodiments of the present invention bring at least the following beneficial effects:

[0049] This invention provides a method, system, and electronic device for detecting cue word attacks using a large language model. The method includes: identifying text blocks to be processed in the original text; performing heuristic semantic parsing on the text blocks to obtain a first suspicious text block and a first risk score for the first suspicious text block; performing semantic analysis on the first suspicious text block and its context information using a pre-trained first language model to obtain a second suspicious text block and a second risk score for the second suspicious text block; performing structured verification of the attack intent of the second suspicious text block based on preset meta-cue words using a pre-trained second language model to obtain an attack intent verification result; wherein the attack intent verification result includes a third suspicious text block and a third risk score for the third suspicious text block; and performing a target response action on the original text based on the first risk score, the second risk score, and the third risk score.

[0050] This technical solution constructs a three-tiered, progressive detection method, effectively covering the entire process from rapid initial screening to deep semantic analysis and attack intent verification. First, heuristic semantic parsing serves as the initial screening step, enabling the preliminary filtering of large-scale text blocks with minimal overhead, effectively ensuring basic system throughput performance and significantly reducing the computational cost of subsequent deep analysis, thus guaranteeing processing efficiency. Second, a first language model performs context-aware semantic analysis on the first suspicious text blocks, identifying medium-complexity attacks disguised by language and accurately filtering high-risk second suspicious text blocks, improving detection accuracy. Finally, a second language model combined with meta-hints is used for structured verification of attack intent, ensuring no complex and covert attacks are missed, achieving accurate identification and interpretable decision-making for novel and complex attack intents. This multi-stage collaborative detection scheme forms a defense-in-depth system, maintaining system real-time performance through rapid front-end filtering and ensuring detection accuracy through deep back-end analysis, avoiding single-dimensional misjudgments and making target response actions more scientific. Therefore, this solution can both meet the needs of efficient processing of long texts and improve the detection accuracy of complex attacks, providing a reliable guarantee for the secure application of large language models.

[0051] Other features and advantages of the invention will be set forth in the following description, or some features and advantages may be inferred from the description or determined without doubt, or may be learned by practicing the techniques described above.

[0052] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0053] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0054] Figure 1 A flowchart illustrating a cue word attack detection method for a large language model provided in this embodiment of the invention;

[0055] Figure 2 A schematic diagram illustrating the workflow of a cue word attack detection method for a large language model provided in an embodiment of the present invention;

[0056] Figure 3 This is a schematic diagram of the structure of a prompt word attack detection system for a large language model provided in an embodiment of the present invention;

[0057] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0058] icon:

[0059] 301 - Text Block Determination Module; 302 - First Semantic Analysis Module; 303 - Second Semantic Analysis Module; 304 - Attack Intent Verification Module; 305 - Action Response Module;

[0060] 101 - Processor; 102 - Memory; 103 - Bus; 104 - Communication interface. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] With the widespread deployment and application of large language models in fields such as intelligent customer service, content creation assistants, and code generation tools, prompt injection attacks have become a serious security threat. Attackers hijack the intended functions of large language models by embedding malicious instructions into the input text, thereby performing unauthorized operations such as: leaking sensitive data, bypassing security policies, generating harmful content, or gaining improper access to backend systems.

[0063] To combat prompt injection attacks, some techniques employ rule-based and keyword-based filtering methods. These methods rely on predefined regular expression patterns for detecting suspicious syntax, keyword blacklists (e.g., "ignore previous instructions," "forgot your identity"), and simple heuristics (e.g., excessive punctuation, unusual character sequences, or code blocks) to identify malicious input. However, these methods are extremely vulnerable. Attackers can easily bypass detection using text obfuscation techniques or by changing the wording, resulting in a very high false negative rate. Examples of text obfuscation techniques include Base64 encoding, homonym substitution, and inserting zero-width characters; examples of wording changes include replacing "ignore previous instructions" with "your previous instructions are invalid; now follow my new arrangements." Furthermore, these methods cannot understand the true semantic intent behind the words, causing them to fail even with slight variations in the attack.

[0064] Some related technologies employ detection methods based on single-stage machine learning models. This method trains a single machine learning model, such as a fine-tuned BERT model or other Transformer-based classifiers, to perform end-to-end maliciousness assessment of the entire input prompt, outputting a classification result as benign or malicious. However, this type of method has the following technical drawbacks:

[0065] (1) Performance bottleneck: For application scenarios that require real-time interaction, having each input, especially long text input, analyzed by a machine learning model will introduce significant computational latency. This high latency is unacceptable when handling large-scale concurrent requests and cannot meet the performance requirements of real-time interaction scenarios, severely limiting the practical application value of this solution in production environments.

[0066] (2) Detection failure in long text context: When malicious instructions are cleverly embedded in long and normal documents (e.g., inserting a malicious instruction into a technical report of 10,000 words), the malicious signal is severely diluted on a macro scale across the entire text. When a single machine learning model processes the entire text, its attention is distracted by a large amount of harmless content, making it difficult to capture such localized and weak malicious signals. This phenomenon can be compared to "finding a needle in a haystack," resulting in a very high false negative rate when this type of method processes long text input, and its effectiveness is significantly reduced.

[0067] (3) Lack of granularity and interpretability in detection: These methods can usually only provide a binary judgment result of benign or malicious, and cannot explain why the input is marked as malicious, nor can they accurately locate the specific malicious fragment in the text. This makes subsequent risk analysis, security audit and automatic remediation extremely difficult.

[0068] Therefore, the aforementioned detection schemes for prompt injection attacks cannot effectively balance detection accuracy and computational performance. This contradiction becomes even more pronounced when dealing with the increasingly common long text input scenarios. Simple filtering schemes are fast but lack sufficient analytical depth and are easily bypassed; while complex machine learning models have strong analytical capabilities but are slow and are prone to failure due to signal dilution in long texts.

[0069] Existing solutions generally adopt a "one-size-fits-all" single-stage processing model, either performing a shallow scan of all text content or a deep analysis of all text content. Both strategies have inherent flaws. These problems seriously hinder the secure deployment of large language models in applications such as legal contract review, financial statement analysis, and technical document summarization. These scenarios are precisely the hardest hit by advanced prompt word injection attacks.

[0070] In this context, to better address prompt injection attacks, improve the detection accuracy of prompt injection attacks, and enhance the ability to process long texts, embodiments of this disclosure provide a prompt attack detection method, system, and electronic device for large language models.

[0071] To facilitate understanding of this embodiment, a method for detecting prompt word attacks in a large language model, as disclosed in this embodiment, will first be described in detail. This method is applied to scenarios involving the detection of prompt word attacks in large language model applications. Figure 1 As shown, the method for detecting cue word attacks using a large language model may include the following steps.

[0072] Step S101: Determine the text block to be processed in the original text; wherein the text block is marked with position information.

[0073] In this embodiment, the user can input raw text of any length, such as technical documents, legal contracts, and financial statements. The user then initiates a text detection request carrying the raw text. The system, which implements a large language model for detecting prompt word attacks, responds to the user's text detection request by intercepting it and obtaining the raw text to be detected for prompt word attacks.

[0074] A series of standardized cleaning operations are performed on the original text to obtain the target text, thereby unifying and purifying the text, eliminating confusion and interference, providing standard and reliable text input for subsequent detection, and improving the accuracy and robustness of prompt word attack detection.

[0075] Then, if the standardized target text is short, less than the preset length threshold, it can be directly used as a text block to be processed. If the standardized target text is long, not less than the preset length threshold, it can be segmented using semantic segmentation techniques. These semantic segmentation techniques do not use a fixed window size for brute-force segmentation; instead, they identify the boundaries where semantics change and segment based on those boundaries. The resulting text blocks, segmented in this way, vary in size and each possesses relatively independent semantics.

[0076] To ensure contextual continuity, this embodiment embeds configurable-length context-overlapping text segments at the beginning and end of each segmented text block when segmenting the target text using semantic-based segmentation techniques, resulting in the final text block. This configuration of context-overlapping text segments ensures that adjacent text blocks overlap in their boundary regions, maintaining contextual continuity and preventing semantic fragmentation at block boundaries.

[0077] Each text block can also carry metadata, which includes at least the location information of the text block in the original text, which can facilitate subsequent risk location and reporting.

[0078] Step S102: Perform heuristic semantic parsing on the text block to obtain the first suspicious text block and the first risk score of the first suspicious text block.

[0079] After obtaining the text block, the first stage of prompt word attack detection is entered, where heuristic semantic parsing is performed on the text block to obtain the first suspicious text block and the first risk score of the first suspicious text block.

[0080] In this embodiment, at least one of the following methods can be used: regular expression library, keyword dictionary, and information entropy detection, to perform heuristic semantic parsing on text blocks and determine a low-level risk score for each text block. If the risk score of a text block is not greater than a preset score threshold (the preset score threshold is, for example, 0 points, i.e., ≤0), the text block can be initially determined to be safe; if the risk score of a text block is greater than the preset score threshold (i.e., >0), the text block can be determined as the first suspicious text block, and its risk score is the first risk score.

[0081] The first suspicious text block and its context information are passed to the second stage described in step S103 for in-depth analysis.

[0082] The above embodiments perform high-speed, low-cost heuristic scanning of text blocks, aiming to quickly identify and mark obvious and known malicious patterns. They can quickly identify the first suspicious text block with extremely low overhead, significantly improve system throughput, focus limited computing resources on subsequent deeper risk content analysis, and accurately focus suspicious text content for subsequent in-depth analysis.

[0083] Step S103: Using a pre-trained first language model, perform semantic analysis on the first suspicious text block and its contextual information to obtain the second suspicious text block and the second risk score of the second suspicious text block.

[0084] After obtaining the first suspicious text block from the first stage output, the second stage, cue word attack detection, proceeds. Using a pre-trained first language model, semantic analysis is performed on the first suspicious text block and its contextual information to obtain a second suspicious text block and a second risk score for each. The goal of this second stage is to understand the true intent hidden behind the first suspicious text block in order to detect complex attacks carried out through language spoofing.

[0085] In this embodiment, the first language model can be, for example, a Transformer-based model trained using cue word samples labeled with classification tags. This first language model can function as a multi-class classifier, performing semantic analysis on the category of the first suspicious text block based on its location and context information, determining multiple classification tags and the confidence level of each tag for each suspicious text block. Classification tags can include: benign, instructional hijacking, data exfiltration attempt, role-playing attack, etc.

[0086] The first suspicious text block may be detected with multiple classification labels. In this embodiment, the classification label with the highest confidence level is used as the target classification label. If the target classification label of the first suspicious text block is a non-benign label, that is, a classification label indicating an attack risk such as instruction hijacking, data theft attempt, or role-playing attack, then the first suspicious text block is identified as the second suspicious text block.

[0087] The second risk score is determined based on the target confidence level corresponding to the target classification label of the second suspicious text block. For example, different confidence levels correspond to different risk scores. Based on the correspondence between confidence levels and risk scores, the second risk score corresponding to the target confidence level is determined.

[0088] The second suspicious text block and its context information are passed to the third stage described in step S104 for verification of the attack intent.

[0089] The above embodiments use a first language model to perform semantic analysis on the first suspicious text block and its context information, enabling deep semantic understanding of the first suspicious text block, effectively identifying covert and variant attacks, thereby significantly reducing false negatives and improving detection accuracy.

[0090] Step S104: Using a pre-trained second language model, the attack intent of the second suspicious text block is structurally verified based on preset meta-prompt words to obtain the attack intent verification result; wherein, the attack intent verification result includes: the third suspicious text block and the third risk score of the third suspicious text block.

[0091] After obtaining the second suspicious text block from the second stage output, the third stage, cue word attack detection, begins. Using a pre-trained second language model, the attack intent of the second suspicious text block is structurally verified based on preset meta-cue words, yielding the attack intent verification result. This third stage utilizes a powerful second language model to simulate the reasoning process of a security analyst, providing a final, expert-level ruling on the second suspicious text block identified in the second stage.

[0092] In this embodiment, the second language model can be a large language model, such as GPT-4 or the Claude series models. The second language model performs structured verification of the attack intent of the second suspicious text block based on preset meta-prompts, guiding questions, the second suspicious text block, and its contextual information, obtaining an attack intent verification result. This result includes: a third suspicious text block, a third risk score for the third suspicious text block, and the reasoning behind the decision for the third suspicious text block.

[0093] The above embodiments introduce a second language model as an adjudicator, combined with meta-cue words, to perform expert-level intent verification on high-risk second suspicious text blocks. In this case, with extremely low false positives, it can accurately identify novel, complex, and deep obfuscation attacks and output interpretable attack intent verification results, greatly reducing false interceptions and improving robustness and reliability in open environments.

[0094] Step S105: Based on the first risk score, the second risk score, and the third risk score, perform the target response action on the original text.

[0095] In this embodiment, suspicious text blocks and risk scores are statistically analyzed in the three stages of the aforementioned prompt word attack detection. A configurable weighted algorithm is used to sum the first, second, and third risk scores to determine the target risk score for the entire original text. Based on the target risk score, a target response action is determined to be performed on the original text. The target response action may include: allowing passage, marking, blocking, or generating a report.

[0096] To better understand the embodiments of this disclosure, the following provides a more detailed description of each step in the prompt word attack detection method for large language models.

[0097] Regarding step S101, which involves determining the text block to be processed in the original text, this embodiment may include:

[0098] First, the original text is standardized to obtain the target text.

[0099] In a specific embodiment, a series of standardized cleaning operations are performed on the original text, including but not limited to: unifying character encoding to Unicode, removing non-printable control characters, and decoding any detected encoding formats (such as URL encoding and Base64 encoding) to restore the original form of the text. Through these standardization processes, the target text is obtained.

[0100] This embodiment standardizes the original text, transforming it into a clean and consistent form. This effectively eliminates bypass methods used by attackers, such as character obfuscation and format transformation, providing a reliable analytical basis for subsequent detection and significantly improving robustness against evasion attacks.

[0101] Secondly, if the length of the target text is less than a preset length threshold, the target text is identified as a text block. If the length of the target text is not less than the preset length threshold, the target text is semantically segmented to obtain text blocks.

[0102] For long text detection, the current challenge lies in efficiently and accurately identifying localized malicious instructions embedded within large amounts of benign text, and overcoming detection failures due to signal dilution. To address this issue, in this embodiment, the core intelligent segmentation algorithm is activated when the target text length is not less than a preset length threshold (e.g., 512 tokens). This intelligent segmentation algorithm (i.e., semantic-based segmentation technology) does not brute-force segment the target text using a fixed window size. Instead, it employs a semantic-based segmentation technique to divide the target text into semantically independent text blocks that contain contextual information and are easy for downstream modules to analyze.

[0103] In one embodiment where semantic segmentation of the target text is performed to obtain text blocks, the following may be included:

[0104] Using a pre-defined sentence converter model, a vector embedding is generated for each sentence in the target text; the cosine distance between the vector embeddings of every two adjacent sentences is determined; the semantic boundary of the sentence changes is determined based on the cosine distance; the target text is segmented into text blocks based on the sentence boundaries; wherein, adjacent text blocks include overlapping text of a first pre-defined length.

[0105] Specifically, a lightweight sentence converter model can be used to generate vector embeddings for each sentence in the target text. The cosine distance between the vector embeddings of any two adjacent sentences is then calculated. It can be understood that the magnitude of the cosine distance represents the magnitude of the semantic change between two consecutive sentences.

[0106] Based on this, if the cosine distance is greater than a preset distance threshold, it can be determined that a significant semantic change has occurred between the two sentences, such as at paragraph or topic transition points. Therefore, these two sentences can be used as the sentence boundaries for semantic segmentation, and the target text can be segmented at these boundaries.

[0107] To ensure the continuity of context, when segmenting the target text at statement boundaries, overlapping text segments of a first preset length are embedded at the beginning and end of each segmented text block, such as overlapping one or two statements, to obtain the final text block. In other words, the final generated text blocks will contain overlapping areas of configurable length.

[0108] Each generated text block will be accompanied by metadata, which includes at least the exact location of the text block in the original text, for subsequent risk identification and reporting.

[0109] To address the issue of long text detection failure, this embodiment abandons the traditional approach of simple, fixed-length segmentation of long texts and instead employs a novel intelligent segmentation and context-preserving mechanism. This embodiment first uses cosine distance to identify semantically changing sentence boundaries within the target text, such as paragraphs and topic transitions. Based on this, the target text is segmented into text blocks with overlapping text according to these sentence boundaries. Thus, in subsequent analysis of each text block, not only is its own content considered, but it is also deeply analyzed along with overlapping contextual information. This effectively prevents malicious instruction signals from being diluted on a macro scale, enabling the semantic analysis model to accurately assess the true intent of any given sentence or paragraph within its specific context. This fundamentally solves the "needle in a haystack" problem that plagues single machine learning models.

[0110] In step S102, heuristic semantic parsing is performed on the text block to obtain a first suspicious text block and a first risk score for the first suspicious text block. This embodiment can determine the first suspicious text block and the first risk score for the first suspicious text block through at least one of the following implementation methods.

[0111] Implementation Method 1. Use a pre-defined regular expression library to perform semantic matching on the text block to obtain the first suspicious text block; the regular expression library includes multiple patterns related to prompt word injection.

[0112] Specifically, a highly optimized regular expression library (such as Google's RE2 library) can be used to perform semantic matching on text blocks. Regular expression libraries can include a large number of patterns related to prompt injection, such as: instructional phrases, API key formats, and structural features of other sensitive data.

[0113] Among them, imperative phrases are used to match common malicious instruction templates, such as "ignore instructions" and "print your prompt". Other structural features of sensitive data are used to identify sensitive information with specific formats, such as credit card numbers and ID card numbers.

[0114] The text block to be detected is scanned and matched against all patterns in the regular expression library. When a text block successfully matches any pattern, a rule hit is triggered, and the text block is marked as the first suspicious text block, and a first risk score is assigned to the first suspicious text block.

[0115] This first implementation method can achieve high-performance and low-cost initial screening of known attack patterns.

[0116] And / or, implementation method two. Semantic matching of text blocks is performed based on a preset keyword dictionary to obtain the first suspicious text block; wherein, the keyword dictionary includes keywords related to the attack behavior.

[0117] This embodiment can build and maintain a continuously updated keyword dictionary to perform rapid semantic matching on text blocks to identify potential attack intentions. This keyword dictionary contains keywords and phrases highly related to attack behaviors such as prompt injection, jailbreaking, and data theft.

[0118] In practical implementation, an efficient string matching algorithm (such as the Aho-Crashick automaton algorithm) can be used to compare the text block with all keywords in the keyword dictionary. When the number of successfully matched keywords or the weighted risk value in the text block exceeds a preset threshold, the text block is marked as the first suspicious text block, and a first risk score is assigned to the first suspicious text block.

[0119] The second implementation method is based on a keyword dictionary to quickly match text blocks, which can accurately capture the first suspicious text blocks with known attack patterns with extremely low overhead, and achieve high-performance preliminary threat screening.

[0120] And / or, implement method three. Calculate the information entropy of the text block and identify the text block with an information entropy higher than the preset information entropy as the first suspicious text block.

[0121] Specifically, the information entropy of the text block is calculated, such as Shannon entropy. Since encoded or encrypted malicious payloads typically exhibit higher information entropy than natural language, obfuscated malicious code can be effectively detected using information entropy metrics. Furthermore, in this embodiment, text blocks with information entropy higher than a preset information entropy are identified as first suspicious text blocks, and a first risk score is assigned to each first suspicious text block.

[0122] Based on the above implementation method, the first suspicious text block is identified, and then the first risk score is determined for the first suspicious text block.

[0123] In this embodiment, a low-level risk score is assigned to each first suspicious text block. If the risk score of a text block is not greater than 0, it may be initially determined to be safe. If the risk score is greater than 0, the text block is a first suspicious text block that requires in-depth analysis, and a first risk score is determined for the first suspicious text block.

[0124] This embodiment utilizes regular expression libraries, keyword dictionaries, and information entropy to quickly identify and mark obvious and known first suspicious text blocks, significantly improving system throughput. This allows limited computing resources to be focused on subsequent, more in-depth risk content analysis, and enables precise targeting of suspicious text content for further in-depth analysis.

[0125] For step S103, the first suspicious text block and its context information are semantically analyzed by the pre-trained first language model to obtain the second suspicious text block and the second risk score of the second suspicious text block. This embodiment may include the following steps (1)-(4).

[0126] (1) Obtain a pre-trained first language model; wherein the first language model is a model based on the Transformer architecture and is trained using prompt word samples labeled with classification tags.

[0127] Specifically, the first language model can be a fine-tuned language model based on the Transformer architecture. To balance performance and accuracy, this embodiment can choose a distilled version of the model optimized for classification tasks (such as DistilledBERT or RoBERTa) or a model specifically designed for understanding tasks (such as DeBERTa).

[0128] The first language model was trained under supervision using a massive number of labeled prompt words; the labels included: benign, instruction hijacking, data theft attempt, role-playing attack, etc.

[0129] Based on the above prompt word samples, the first language model is trained as a multi-class classifier, capable of assigning fine-grained classification labels to each input text block.

[0130] (2) Input the first suspicious text block and its context information into the first language model; wherein, the context information of the first suspicious text block includes: the position information of the first suspicious text block in the original text, and the second preset length of overlapping text between the first suspicious text block and the adjacent text block.

[0131] To make a more accurate judgment, the input of the first language model includes not only the first suspicious text block to be analyzed, but also metadata such as its location information, as well as overlapping text of a second preset length from adjacent text blocks.

[0132] (3) Based on the first suspicious text block and its context information, the first language model is used to determine the target classification label and the second risk score of the target classification label corresponding to the first suspicious text block.

[0133] The first language model determines the confidence level of each classification label corresponding to the first suspicious text block based on the first suspicious text block and its context information. The classification label with the highest confidence level is determined as the target classification label, and this highest confidence level can be quantified as a second risk score. This second risk score not only reflects the confidence level of the first language model in the current classification result, but also provides a fine-grained risk assessment basis for downstream decision-making and response.

[0134] (4) The first suspicious text block with a second risk score higher than the preset score value is identified as the second suspicious text block. The second suspicious text block is sent to the third stage for verification of the attack intent.

[0135] This embodiment improves the detection accuracy against complex, covert, and semantically suggestive injection attacks by introducing a specially trained first language model and combining it with contextual information. First, the first language model not only analyzes the first suspicious text block itself but also integrates its positional information within the original text and overlapping text with adjacent blocks, effectively reconstructing the local context. This enables the first language model to understand malicious instruction logic across sentences and paragraphs, accurately identifying advanced attacks that are diluted or disguised within long texts.

[0136] Secondly, by outputting specific target classification labels and a second risk score, this embodiment can not only make a binary judgment but also reveal the nature and intent of the attack, providing a more valuable decision-making basis for security audits and subsequent responses.

[0137] In addition, this embodiment only performs in-depth analysis on the first suspicious text block in the first stage, which involves a small amount of data and avoids the performance overhead of feeding all the original text into a complex model. Thus, without significantly increasing the system latency, it achieves a significant leap in detection depth and breadth.

[0138] Current prompt injection detection solutions mostly adopt a single technical approach, resulting in low accuracy in detecting complex attacks: rule-based heuristic methods have poor generalization ability for new, variant, and semantic attacks, while pure semantic models have low detection efficiency for known patterns, making it difficult to build a comprehensive defense system.

[0139] To address this issue and reliably detect semantically complex, context-hidden, or deeply obfuscated prompt injection attacks, the embodiments described in steps S102 and S103 do not rely on any single detection method. Instead, they employ a hybrid analysis approach, organically combining lightweight heuristic semantic parsing with semantic analysis based on a first language model. Heuristic semantic parsing operates in a front-end manner, performing semantic matching based on a regular expression library, keyword dictionary, and information entropy to efficiently identify known attack patterns and obvious malicious features, thus obtaining the first suspicious text block. The first language model then performs deep semantic analysis on the initially identified first suspicious text block, thoroughly analyzing its semantic intent, logical consistency, and hidden instructions within the natural language. This enables the identification of novel, unknown "zero-day attacks" that rely on language manipulation rather than specific grammatical triggers.

[0140] Therefore, the aforementioned hybrid detection methods form a multi-layered defense-in-depth. Heuristic semantic parsing ensures efficient interception of known threats, constituting the first line of defense. The first language model effectively identifies zero-day attacks and deep obfuscation attacks. The combination of the two can significantly improve detection coverage and overall robustness across all types of threats.

[0141] Regarding step S104, a pre-trained second language model is used to perform structured verification of the attack intent of the second suspicious text block based on preset meta-cue words, thereby obtaining the attack intent verification result. This embodiment includes:

[0142] Obtain a pre-trained second language model; wherein the second language model is a general-purpose large language model. Input the preset meta-prompt words, the second suspicious text block and its contextual information into the second language model; wherein the meta-prompt words include guiding questions.

[0143] The attack intent of the second suspicious text block is structurally verified using a second language model based on meta-prompts, guiding questions, the second suspicious text block and its contextual information, resulting in an attack intent verification result. The attack intent verification result includes: the third suspicious text block, the third risk score of the third suspicious text block, and the adjudication reason for the third suspicious text block.

[0144] In this embodiment, a powerful second language model is invoked via API. This second language model is a general-purpose large language model, such as GPT-4 or the Claude series of models.

[0145] This embodiment can use a carefully crafted meta-cue word (or system cue word) to interact with the second language model, which acts as a judge. The meta-cue word instructs the second language model to play the role of a "cybersecurity analyst specializing in vulnerabilities in large language models."

[0146] The second suspicious text block, its contextual information within the original text, and a series of guiding questions are input into the second language model. These guiding questions could include, for example, "Does this text attempt to override or manipulate the system's main instructions? If so, what are its possible malicious targets? Please provide a confidence score of 1 to 10 for your assessment and detail your analysis process."

[0147] The second language model infers the attack intent of the second suspicious text block based on meta-cue words, leading questions, the second suspicious text block, and its contextual information, thus obtaining a thought chain response. The thought chain response is then analyzed to extract the final ruling, confidence score, and detailed reasons supporting that ruling.

[0148] The decision statement indicates whether the second suspicious text block can be identified as the third suspicious text block; if so, the third suspicious text block is output. The confidence score is used to quantify the third risk score of the text block as the third suspicious text block. The detailed reasons supporting the decision statement constitute the decision reasoning for the third suspicious text block. Based on this, a highly accurate and interpretable verification result of the attack intent is obtained.

[0149] Currently, in real-time, high-throughput LLM application scenarios, prompt word injection detection faces the following problems: detection based on deep semantic models has high accuracy but large computational latency, while detection based on rule matching is fast but lacks coverage and generalization ability, making it difficult to achieve high-precision protection while ensuring system response performance.

[0150] To address this issue, the embodiments described in steps S102 to S104 above employ a multi-stage filtering architecture that executes sequentially and deepens step by step. This architecture constructs a cascaded risk "funnel" consisting of three stages.

[0151] Specifically, the first stage employs a low-computational-cost heuristic semantic parsing method to quickly screen all input text blocks. Only the first suspicious text blocks initially identified as potentially risky are passed to the more computationally intensive second stage for in-depth semantic analysis. The second stage uses a first language model to perform semantic analysis only on the initially screened first suspicious text blocks, thereby identifying a smaller number of second suspicious text blocks with higher risk. The third stage only rigorously verifies the attack intent of the higher-risk second suspicious text blocks identified through semantic analysis.

[0152] This cascaded, multi-stage risk detection architecture, through an intelligent resource allocation mechanism, ensures that most (typically over 99%) of benign inputs are quickly allowed through with extremely low latency at the front end, while valuable deep learning resources are precisely focused on the very few truly risky and suspicious text blocks. This significantly optimizes overall throughput at the system level, achieving a balance between high detection accuracy and low processing latency, effectively guaranteeing real-time security protection in high-concurrency scenarios, and meeting the needs of real-time detection.

[0153] Regarding step S105, based on the first risk score, the second risk score, and the third risk score, a target response action is performed on the original text. This embodiment may include:

[0154] The first risk score, second risk score, and third risk score are weighted to obtain the target risk score of the original text; the target response action corresponding to the target risk score is determined based on the correspondence between the risk scores and response actions; and the target response action is executed on the original text.

[0155] This embodiment gathers and integrates risk scores and suspicious text blocks from all preceding stages, calculates a final risk score for the original text as a whole, and determines the appropriate response action based on this score.

[0156] In one implementation, a configurable weighted algorithm can be used to fuse the first, second, and third risk scores. For example, the weight of the first risk score from the first stage is lower than the weight of the second risk score from the second stage, and the weight of the second risk score from the second stage is lower than the weight of the third risk score from the third stage. This weighting configuration indicates that the more suspicious text blocks are confirmed through subsequent, more complex analysis stages, the higher the confidence and value of their detection results. Therefore, when weighting the first, second, and third risk scores, assigning higher weights to the risk scores of subsequent stages essentially allows suspicious text blocks with higher confidence and greater decision value to dominate the final decision, thereby making the final target risk score more accurate and reliable.

[0157] Then, the first risk score, second risk score, and third risk score are weighted and scored to obtain the final target risk score of the original text. Furthermore, based on the correspondence between risk scores and response actions, the target response action corresponding to the target risk score is determined. The correspondence between risk scores and response actions can be seen in Table 1 below:

[0158] Table 1: Correspondence between risk scores and response actions

[0159]

[0160] According to the correspondence between the above risk scores and response actions, when the target risk score is lower than the threshold $T_1$, the original text is determined to be safe, and it can be released and passed to the target LLM.

[0161] When the target risk score is between the thresholds $T_1$ and $T_2$ ($T_1\leq\text{Score}<T_2$), the original text is allowed to be passed to the target LLM, but at the same time, a mark is executed to generate a security alert log for manual review.

[0162] When the target risk score is higher than or equal to the threshold $T_2$, the original text is intercepted, and an error or warning message is returned to the user. Alternatively, in this embodiment, the original text can also be disinfected, which can be understood as: removing the detected third suspicious text block with malicious content, or removing the first suspicious text block, the second suspicious text block, and the third suspicious text block. After that, the remaining text after the disinfection operation is passed to the target LLM.

[0163] Based on the above embodiments, for any original text that is marked or intercepted, a detailed JSON - formatted report can be generated, which includes the final decision (i.e., the first suspicious text block, the second suspicious text block, and the third suspicious text block), the target risk score, and the detailed findings from each analysis stage.

[0164] Refer to Figure 2 , in a practical application scenario, the method for detecting prompt - word attacks of the large - language model provided by the present disclosure can be applied to an end - to - end workflow, as shown below:

[0165] Start: When the user submits the original text to the target large - language model through the application on the user side, the user side initiates a text detection request carrying the original text to the target large - language model.

[0166] Request Interception: In response to the text detection request from the user side, intercept the text detection request and obtain the original text to be detected for prompt - word attacks. Specifically, the text detection request is intercepted by the middleware to obtain the original text carried by the text detection request.

[0167] Pre - processing and Chunking: Perform standardization processing on the original text to obtain the target text; if the length of the target text is not less than the preset length threshold, perform semantic segmentation on the target text to obtain text chunks containing overlapping text.

[0168] The first stage of scanning involves heuristic semantic analysis of the text blocks to obtain the first suspicious text block and its first risk score. Specifically, all text blocks are sent to a rapid heuristic scanning stage, where the vast majority of benign content is quickly identified and its suspicions are eliminated, with the first risk score for benign content being no greater than 0; while the first suspicious text blocks with a first risk score greater than 0 are passed to the next stage.

[0169] The second stage of analysis involves using a pre-trained first language model to perform semantic analysis on the first suspicious text block and its contextual information, resulting in a second risk score for the second suspicious text block. Specifically, the first suspicious text block from the first stage undergoes deep analysis by the context-aware semantic analysis first language model to obtain its second risk score for this stage. If this second risk score is lower than a preset score, it is classified as low-risk, and the low-risk first suspicious text block is removed from the list of suspects. If the second risk score is not lower than the preset score, it is classified as high-risk, and the high-risk first suspicious text block is then passed on to the next stage as the second suspicious text block.

[0170] The third stage of verification involves using a pre-trained second language model to perform structured verification of the attack intent of the second suspicious text block based on preset meta-cue words, thus obtaining the attack intent verification result. Specifically, the second suspicious text block with the highest risk from the second stage is sent to the third stage for attack intent verification, where a powerful second language model makes the final, interpretable decision.

[0171] Integrated Decision-Making and Action Response: Based on the first, second, and third risk scores, the system executes targeted response actions on the original text. Specifically, it collects all suspicious text blocks and risk scores from the entire processing pipeline and calculates the final target risk score for the original text using a weighted algorithm. Based on the target risk score, it executes the corresponding target response action on the original text, such as allowing passage, marking, or blocking, and generates corresponding response messages and detailed analysis reports.

[0172] End: The processing flow has ended. Thus, this embodiment has successfully protected the backend large language model from potential attacks and securely delivered a benign request.

[0173] To more clearly demonstrate the differences between the prompt word attack detection method for large language models provided in this disclosure and related technologies, the key performance indicators of different technical solutions are compared in Table 2 below:

[0174] Table 2: Technology Comparison

[0175]

[0176] In summary, the large language model-based cue word attack detection method provided in this embodiment of the invention includes: determining the text block to be processed in the original text; performing heuristic semantic parsing on the text block to obtain a first suspicious text block and a first risk score for the first suspicious text block; performing semantic analysis on the first suspicious text block and its context information using a pre-trained first language model to obtain a second suspicious text block and a second risk score for the second suspicious text block; performing structured verification of the attack intent of the second suspicious text block based on preset meta-cue words using a pre-trained second language model to obtain an attack intent verification result; wherein the attack intent verification result includes a third suspicious text block and a third risk score for the third suspicious text block; and performing a target response action on the original text based on the first risk score, the second risk score, and the third risk score.

[0177] This technical solution constructs a three-tiered, progressive detection method, effectively covering the entire process from rapid initial screening to deep semantic analysis and attack intent verification. First, heuristic semantic parsing serves as the initial screening step, enabling the preliminary filtering of large-scale text blocks with minimal overhead, effectively ensuring basic system throughput performance and significantly reducing the computational cost of subsequent deep analysis, thus guaranteeing processing efficiency. Second, a first language model performs context-aware semantic analysis on the first suspicious text blocks, identifying medium-complexity attacks disguised by language and accurately filtering high-risk second suspicious text blocks, improving detection accuracy. Finally, a second language model combined with meta-hints is used for structured verification of attack intent, ensuring no complex and covert attacks are missed, achieving accurate identification and interpretable decision-making for novel and complex attack intents. This multi-stage collaborative detection scheme forms a defense-in-depth system, maintaining system real-time performance through rapid front-end filtering and ensuring detection accuracy through deep back-end analysis, avoiding single-dimensional misjudgments and making target response actions more scientific. Therefore, this solution can both meet the needs of efficient processing of long texts and improve the detection accuracy of complex attacks, providing a reliable guarantee for the secure application of large language models.

[0178] Corresponding to the above method embodiments, this invention provides a prompt word attack detection system for large language models, such as... Figure 3 As shown, the system includes the following modules: a text block determination module, a first semantic analysis module, a second semantic analysis module, an attack intent verification module, and an action response module. Among them:

[0179] The text block determination module 301 is used to determine the text blocks to be processed in the original text; wherein, the text blocks are marked with position information;

[0180] The first semantic analysis module 302 is used to perform heuristic semantic analysis on the text block to obtain the first suspicious text block and the first risk score of the first suspicious text block;

[0181] The second semantic analysis module 303 is used to perform semantic analysis on the first suspicious text block and its context information using a pre-trained first language model to obtain the second suspicious text block and the second risk score of the second suspicious text block.

[0182] The attack intent verification module 304 is used to perform structured verification of the attack intent of the second suspicious text block based on preset meta-prompt words using a pre-trained second language model, and obtain the attack intent verification result; wherein, the attack intent verification result includes: the third suspicious text block and the third risk score of the third suspicious text block;

[0183] Action response module 305 is used to perform target response actions on the original text based on the first risk score, the second risk score and the third risk score.

[0184] The large language model-based prompt attack detection system provided in this embodiment is implemented using a modular, pipelined architecture to ensure high cohesion and low coupling, facilitating maintenance and expansion. This system architecture design allows data to flow sequentially through each detection stage, with each module performing highly specialized analysis tasks and passing its results to the next module.

[0185] The large language model prompt attack detection system consists of five core modules: a text block identification module 301, a first-stage semantic analysis module 302, a second-stage semantic analysis module 303, a third-stage attack intent verification module 304, and an action response module 305. This system can be deployed as a security middleware or API (Application Programming Interface) endpoint to intercept and analyze all input requests flowing to the target large language model.

[0186] In some embodiments, the text block determination module 301 further includes:

[0187] A standardization unit is used to standardize the original text to obtain the target text;

[0188] The first text block determination unit is used to determine the target text as a text block if the length of the target text is less than a preset length threshold.

[0189] The second text block determination unit is used to perform semantic segmentation on the target text to obtain the text block if the length of the target text is not less than a preset length threshold.

[0190] In some embodiments, the second text block determining unit is further configured to:

[0191] A vector embedding is generated for each sentence in the target text using a pre-defined sentence converter model;

[0192] Determine the cosine distance between the vector embeddings corresponding to every two adjacent sentences;

[0193] The semantic boundary of the statement undergoing semantic change is determined based on the cosine distance;

[0194] The target text is divided into text blocks based on the statement boundaries; wherein, adjacent text blocks include overlapping text of a first preset length.

[0195] In some implementations, the first semantic analysis module 302 is further configured to:

[0196] The text block is semantically matched using a pre-defined regular expression library to obtain the first suspicious text block; wherein, the regular expression library includes multiple patterns related to prompt word injection;

[0197] And / or, semantic matching of the text block is performed according to a preset keyword dictionary to obtain the first suspicious text block; wherein, the keyword dictionary includes keywords related to the attack behavior;

[0198] And / or, calculate the information entropy of the text block, and identify the text block whose information entropy is higher than a preset information entropy as the first suspicious text block;

[0199] A first risk score is determined for the first suspicious text block.

[0200] In some implementations, the second semantic analysis module 303 is further used for:

[0201] Obtain a pre-trained first language model; wherein the first language model is a model based on the Transformer architecture and is trained using prompt word samples labeled with classification tags;

[0202] The first suspicious text block and its context information are input into the first language model; wherein, the context information of the first suspicious text block includes: the position information of the first suspicious text block in the original text, and the overlapping text of the first suspicious text block and adjacent text blocks of a second preset length;

[0203] The first language model determines the target classification label corresponding to the first suspicious text block and the second risk score of the target classification label based on the first suspicious text block and its context information.

[0204] The first suspicious text block whose second risk score is higher than the preset score value is identified as the second suspicious text block.

[0205] In some implementations, the attack intent verification module 304 is also used for:

[0206] Obtain a pre-trained second language model; wherein, the second language model is a general-purpose large language model;

[0207] The preset meta-prompt words, the second suspicious text block, and their contextual information are input into the second language model; wherein, the meta-prompt words include guiding questions;

[0208] The second language model performs a structured verification of the attack intent of the second suspicious text block based on the meta-prompt words, the guiding questions, the second suspicious text block and its contextual information, to obtain an attack intent verification result; wherein, the attack intent verification result includes: a third suspicious text block, a third risk score of the third suspicious text block and a ruling reason for the third suspicious text block.

[0209] In some implementations, the action response module 305 is also used for:

[0210] The first risk score, the second risk score, and the third risk score are weighted and scored to obtain the target risk score of the original text.

[0211] Based on the correspondence between risk scores and response actions, determine the target response action corresponding to the target risk score;

[0212] Perform the target response action on the original text.

[0213] In some implementations, the large language model cue word attack detection system also includes:

[0214] The interception module is used to respond to the text detection request from the user, intercept the text detection request, and obtain the original text to be detected for prompt word attacks.

[0215] The prompt attack detection system for large language models provided in this embodiment has the same technical features as the prompt attack detection method for large language models provided in the above embodiments, and therefore can solve the same technical problems and achieve the same technical effects. For the sake of brevity, any parts not mentioned in the embodiment section can be referred to the corresponding content in the aforementioned prompt attack detection method embodiment for large language models.

[0216] This embodiment also provides an electronic device, the structural schematic diagram of which is shown below. Figure 4As shown, the device includes a processor 101 and a memory 102; wherein, the memory 102 is used to store one or more computer instructions, which are executed by the processor to implement the above-mentioned large language model prompt attack detection method.

[0217] Figure 4 The electronic device shown also includes a bus 103 and a communication interface 104, with the processor 101, communication interface 104 and memory 102 connected via the bus 103.

[0218] The memory 102 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device. The bus 103 may be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0219] The communication interface 104 is used to connect to at least one user terminal and other network units through a network interface, and to send encapsulated IPv4 packets or IPv4 packets to the user terminal through the network interface.

[0220] Processor 101 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 101 or by instructions in software form. The processor 101 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this disclosure can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 102. The processor 101 reads the information in memory 102 and, in conjunction with its hardware, completes the steps of the method described in the foregoing embodiments.

[0221] This invention also provides a readable storage medium storing a computer program, which, when run by a processor, executes the steps of the prompt word attack detection method for the large language model described in the foregoing embodiments.

[0222] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0223] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0224] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0225] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0226] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A prompt word attack detection method of a large language model, characterized in that, The method comprises the following steps: determining a text block to be processed in the original text; performing heuristic semantic analysis on the text block to obtain a first suspicious text block and a first risk score of the first suspicious text block; performing semantic analysis on the first suspicious text block and its context information by using a pre-trained first language model to obtain a second suspicious text block and a second risk score of the second suspicious text block; performing structured verification of attack intent on the second suspicious text block according to a preset meta prompt word by using a pre-trained second language model to obtain an attack intent verification result; wherein the attack intent verification result comprises a third suspicious text block and a third risk score of the third suspicious text block; performing a target response action on the original text based on the first risk score, the second risk score and the third risk score; the semantic analysis on the first suspicious text block and its context information by using the pre-trained first language model to obtain the second suspicious text block and the second risk score of the second suspicious text block comprises: obtaining a pre-trained first language model; wherein the first language model is a model based on a Transformer architecture and is trained by using prompt word samples labeled with classification labels; inputting the first suspicious text block and its context information into the first language model; wherein the context information of the first suspicious text block comprises position information of the first suspicious text block in the original text and overlapping text of a second preset length between the first suspicious text block and adjacent text blocks; determining, by using the first language model, a target classification label corresponding to the first suspicious text block and a second risk score of the target classification label according to the first suspicious text block and its context information; determining the first suspicious text block with a second risk score higher than a preset score value as a second suspicious text block; the structured verification of attack intent on the second suspicious text block according to the preset meta prompt word by using the pre-trained second language model to obtain the attack intent verification result comprises: obtaining a pre-trained second language model; wherein the second language model is a general large language model; inputting the preset meta prompt word, the second suspicious text block and its context information into the second language model; wherein the meta prompt word comprises a guiding question; performing, by using the second language model, structured verification of attack intent on the second suspicious text block according to the meta prompt word, the guiding question, the second suspicious text block and its context information to obtain an attack intent verification result; wherein the attack intent verification result comprises a third suspicious text block, a third risk score of the third suspicious text block and a ruling reason of the third suspicious text block.

2. The prompt attack detection method of a large language model according to claim 1, characterized in that, the determination of the text block to be processed in the original text comprises: performing standardization processing on the original text to obtain a target text; if the length of the target text is less than a preset length threshold, determining the target text as a text block; If the length of the target text is not less than a preset length threshold, the target text is subjected to semantic segmentation to obtain the text blocks.

3. The prompt attack detection method of a large language model according to claim 2, characterized in that, The semantic segmentation of the target text to obtain the text blocks comprises: generating vector embeddings for each sentence in the target text through a preset sentence converter model; determining the cosine distance between the vector embeddings corresponding to each pair of adjacent sentences; determining the sentence boundary at which a semantic change occurs according to the cosine distance; segmenting the target text into text blocks based on the sentence boundary; wherein the adjacent text blocks include overlapping text of a first preset length. 4.The prompt attack detection method of a large language model according to claim 1, characterized in that, The heuristic semantic analysis of the text blocks to obtain the first suspicious text block and the first risk score of the first suspicious text block comprises: performing semantic matching on the text blocks using a preset regular expression library to obtain the first suspicious text block; wherein the regular expression library includes multiple patterns related to prompt word injection; and / or, performing semantic matching on the text blocks according to a preset keyword dictionary to obtain the first suspicious text block; wherein the keyword dictionary includes keywords related to attack behavior; and / or, calculating the information entropy of the text blocks, and determining the text block with an information entropy higher than a preset information entropy as the first suspicious text block; determining the first risk score of the first suspicious text block.

5. The prompt attack detection method of a large language model according to claim 1, characterized in that, The execution of the target response action on the original text based on the first risk score, the second risk score, and the third risk score comprises: performing weighted scoring on the first risk score, the second risk score, and the third risk score to obtain a target risk score of the original text; determining the target response action corresponding to the target risk score according to the correspondence between risk scores and response actions; executing the target response action on the original text.

6. The prompt attack detection method of a large language model according to claim 1, wherein, Before determining the text block to be processed in the original text, the method further comprises: intercepting a text detection request from a user terminal and obtaining an original text to be subjected to prompt word attack detection in response to the text detection request.

7. A prompt word attack detection system of a large language model, characterized by, The system comprises the following modules: a text block determination module for determining a text block to be processed in an original text; wherein the text block is marked with position information; a first semantic analysis module for performing heuristic semantic analysis on the text block to obtain a first suspicious text block and a first risk score of the first suspicious text block; a second semantic analysis module for performing semantic analysis on the first suspicious text block and its context information through a pre-trained first language model to obtain a second suspicious text block and a second risk score of the second suspicious text block; an attack intent verification module for performing structured verification of attack intent on the second suspicious text block according to a preset meta prompt word through a pre-trained second language model to obtain an attack intent verification result; wherein the attack intent verification result includes a third suspicious text block and a third risk score of the third suspicious text block; An action response module is configured to perform a target response action on the original text based on the first risk score, the second risk score, and the third risk score. The second semantic analysis module is further configured to: obtain a pre-trained first language model; the first language model is a model based on a Transformer architecture and is trained using prompt word samples labeled with classification labels; input the first suspicious text block and its context information into the first language model; the context information of the first suspicious text block includes: position information of the first suspicious text block in the original text, and overlapping text of a second preset length between the first suspicious text block and adjacent text blocks; determine, by the first language model, a target classification label corresponding to the first suspicious text block and a second risk score of the target classification label according to the first suspicious text block and its context information; determine a second suspicious text block by determining that the first suspicious text block has a second risk score higher than a preset score value. The attack intention verification module is further configured to: obtain a pre-trained second language model; the second language model is a general large language model; input a preset meta prompt word, the second suspicious text block and its context information into the second language model; the meta prompt word includes a guiding question; perform, by the second language model, a structured verification of an attack intention of the second suspicious text block according to the meta prompt word, the guiding question, the second suspicious text block and its context information, to obtain an attack intention verification result; the attack intention verification result includes a third suspicious text block, a third risk score of the third suspicious text block, and a ruling reason of the third suspicious text block.

8. An electronic device, comprising: Comprising: A processor and a storage device; the storage device stores a computer program, which realizes the steps of the prompt word attack detection method of the large language model according to any one of claims 1 to 6 when the computer program is run by the processor.

Images