A large model prompt injection attack dynamic adaptive defense method and system

Abstract

The application relates to a large model Prompt injection attack dynamic self-adaptive defense method and system, and relates to the technical field of artificial intelligence security. The method comprises the following steps: performing semantic intention analysis on a to-be-detected text by using a semantic detection model and an attack feature library to determine a risk level, and performing hierarchical attack isolation treatment; if the risk level is high, the to-be-detected text is intercepted; if the risk level is low, the to-be-detected text is released; if the risk level is medium, the to-be-detected text is subjected to semantic reconstruction processing and returned for re-inspection until the risk level becomes low or a preset termination condition is met; log data in the defense process is collected, and after verification, the parameters of the semantic detection model are updated or the attack feature library is updated based on the verified log data. Through deep semantic analysis and hierarchical treatment mechanism, combined with log collection and model self-updating closed loop, the problems of insufficient semantic understanding, static solidification of defense strategy and lack of self-adaptive ability of the prior art are solved, and the safety and adaptability of large model application are significantly improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence security, specifically to a dynamic adaptive defense method and system for large-scale Prompt injection attacks. Background Technology

[0002] With the increasing application of large language models in areas such as government affairs processing, financial risk control, online education, and enterprise intelligent services, model security has become a critical issue. Among them, Prompt injection attacks are a typical attack method targeting large models. Attackers construct malicious input commands to induce the large model to ignore preset system prompts, bypass permission restrictions and content specifications, and then output sensitive information, illegal content, or perform unauthorized operations.

[0003] To defend against Prompt injection attacks, existing technologies mainly employ the following three types of defense methods: The first type is rule-based defense technology. This technology relies on manually preset fixed rules to perform surface-level string checks on the input text, mainly through building a fixed keyword matching library, regular expression filtering rules, and a sensitive word blacklist. In specific implementation, known Prompt injection attack signature words and malicious sentence templates are pre-compiled into a rule library. When a user inputs a Prompt, the system matches the input text against the rule library. If a match is found, the input is blocked; otherwise, it is allowed to proceed.

[0004] The second category is basic text classification defense technology. This technology adds a lightweight classification model and shallow feature extraction to rule-based matching. The implementation process includes preprocessing the input text such as word segmentation, stop word removal, and cleaning; extracting shallow features such as word frequency, sentence length, character distribution, and lexical entropy; and then inputting the features into a lightweight classification model such as Naive Bayes, logistic regression, or a small convolutional neural network to calculate the malicious probability of the input text. If the probability exceeds a preset threshold, the text is blocked.

[0005] The third type is hybrid defense technology. This technology combines the advantages of rule-based and basic text classification defense to build a two-layer detection architecture. The first layer uses rule-based defense to quickly intercept obvious malicious attacks, while the second layer uses a basic text classification model to perform secondary detection on inputs that were not intercepted by the first layer. The results from both layers are combined to determine whether to block or allow the attack.

[0006] However, the aforementioned existing technologies have the following drawbacks when facing advanced Prompt injection attacks: First, there is insufficient understanding of deep semantic intent. Existing technologies mostly focus on surface feature detection, making it difficult to penetrate the textual surface and interpret the underlying instruction logic. For example, when attackers employ paraphrasing, metaphorical manipulation, step-by-step detours, or Base64-encoded malicious instructions and nested within long texts, traditional rule matching or shallow feature extraction methods are highly susceptible to failure. This allows numerous covert attacks to bypass defense systems, posing security vulnerabilities.

[0007] Secondly, defense strategies lag behind attack evolution. The maintenance of existing rule and signature databases relies heavily on manual intervention, resulting in long update cycles. Faced with rapidly evolving attack methods (especially zero-day attacks), defense systems often experience time lags, creating defense vacuums. Furthermore, manual analysis cannot exhaustively cover all attack scenarios, leading to limited defense coverage.

[0008] Secondly, the risk handling mechanism is too rigid. Most existing defense logics follow a black-and-white binary approach: if a risk is identified, it is directly blocked; otherwise, it is allowed. This approach lacks flexibility: while high-risk attacks certainly require blocking, for medium-risk inputs with unclear characteristics or those in a suspected state, direct blocking may inadvertently harm normal business operations and impact user experience, while direct allowing poses security risks. Current technology lacks differentiated handling methods for medium-risk text (such as post-reconstruction review), making it difficult to strike a balance between security and business continuity.

[0009] Finally, there is a lack of a closed-loop mechanism for adaptive evolution. Existing defense systems are typically open-loop, lacking a complete feedback loop from attack log collection and automatic discovery of new features to iterative optimization of the detection model. The system cannot use misjudgment data or new attack samples generated during actual operation to self-correct, causing the defense capability to gradually weaken over time.

[0010] Therefore, how to provide a large-scale Prompt injection attack defense method that can deeply understand semantic intent, has dynamic adaptive update capabilities, and can handle risks in a layered manner has become an urgent problem to be solved in the current technical field. Summary of the Invention

[0011] This application provides a dynamic adaptive defense method and system for large-model Prompt injection attacks, which can solve the technical problems existing in the prior art, such as static and fixed defense strategies, failure to identify hidden attacks, crude risk handling methods, and lack of security closed-loop update capabilities.

[0012] In a first aspect, embodiments of this application provide a dynamic adaptive defense method for large-scale Prompt injection attacks. The method includes: acquiring and preprocessing the original text input by the user to obtain the text to be detected; analyzing the semantic intent of the text to be detected using a semantic detection model and a known attack feature library to obtain a risk probability and determine a risk level; performing tiered processing according to the risk level: if the risk level is high, the original text is blocked; if the risk level is low, the original text is allowed to proceed to the large-scale model for inference; if the risk level is medium, the text to be detected is semantically reconstructed to obtain reconstructed text, and the reconstructed text is returned as a new text to be detected to execute the semantic intent analysis step until the risk level becomes low or a preset termination condition is met; collecting and verifying log data during the defense process, and updating the semantic detection model or the attack feature library based on the verified log data.

[0013] In conjunction with the first aspect, in one implementation, the preprocessing includes: removing garbled characters, redundant spaces, and special symbols from the original text, and unifying capitalization and punctuation; using a standardized dictionary and mapping algorithm to convert abbreviations and colloquialisms in the original text into standard written language; extracting the text length features of the original text, and normalizing the text length features.

[0014] In conjunction with the first aspect, in one implementation, obtaining the risk probability includes: inputting the text to be detected into the semantic detection model to obtain an attack identification probability; converting the text to be detected into a text vector, calculating the similarity between the text vector and a known attack vector in the known attack feature library, and obtaining a similarity probability after mapping processing; and performing a weighted fusion based on the attack identification probability and the similarity probability to obtain the risk probability.

[0015] In conjunction with the first aspect, in one implementation, determining the risk level includes: if the risk probability is greater than or equal to a first preset threshold, determining it as a high-risk level; if the risk probability is less than the first preset threshold but greater than or equal to a second preset threshold, determining it as a medium-risk level; if the risk probability is less than the second preset threshold, determining it as a low-risk level; wherein the first preset threshold is greater than the second preset threshold.

[0016] In conjunction with the first aspect, in one implementation, the semantic reconstruction includes at least one of the following: identifying and removing leading and nested instructions from the text to be detected; converting imperative statements in the text to be detected into declarative statements; and inserting interfering information at key positions in the text to be detected to disrupt the semantic coherence of the attack instructions.

[0017] In conjunction with the first aspect, in one implementation, the preset termination condition includes at least one of the following: the number of re-inspections reaches a preset number threshold; the total time consumed in the re-inspection process exceeds a preset duration threshold; the change in risk probability obtained from two consecutive re-inspections is less than a preset convergence threshold.

[0018] In conjunction with the first aspect, in one implementation, the verification includes: triggering a verification task when the number of suspected new attack feature sets collected exceeds a preset threshold; and confirming that the sample corresponding to the log data is an attack sample through manual review, multi-model voting confirmation, or sandbox environment verification.

[0019] In conjunction with the first aspect, in one implementation, updating the semantic detection model includes: performing regression testing on the fine-tuned model in a sandbox environment, and after confirming that the accuracy of historical scenarios has not decreased, activating the fine-tuned model as the semantic detection model.

[0020] Secondly, embodiments of this application provide a dynamic adaptive defense system for large-scale Prompt injection attacks. The system includes: a preprocessing module for acquiring and preprocessing user-inputted raw text to obtain text to be detected; a detection module for analyzing the semantic intent of the text to be detected using a semantic detection model and a known attack feature library to obtain a risk probability and determine a risk level; a handling module for performing tiered handling based on the risk level: if the risk level is high, the raw text is blocked; if the risk level is low, the raw text is allowed to proceed to large-scale model inference; if the risk level is medium, the text to be detected is semantically reconstructed to obtain reconstructed text, and the reconstructed text is used as a new text to be detected to trigger the detection module for re-inspection until the risk level becomes low or a preset termination condition is met; and an update module for collecting and verifying log data during the defense process, and updating the semantic detection model or the attack feature library based on the verified log data.

[0021] Thirdly, embodiments of this application provide a dynamic adaptive defense device for large-model Prompt injection attacks. The large-model Prompt injection attack dynamic adaptive defense device includes a processor, a memory, and a large-model Prompt injection attack dynamic adaptive defense program stored in the memory and executable by the processor. When the large-model Prompt injection attack dynamic adaptive defense program is executed by the processor, it implements the steps of the large-model Prompt injection attack dynamic adaptive defense method as described in any one of the first aspects.

[0022] The beneficial effects of the technical solutions provided in this application include: By introducing a deep semantic detection and feature library fusion mechanism, the problem of failure in identifying hidden attacks in related technologies is solved. This application's embodiments not only utilize a semantic detection model to analyze text intent but also combine it with a known attack feature library to calculate semantic similarity, obtaining the risk probability through weighted fusion. This dual verification mechanism can effectively identify hidden attacks such as synonym rewriting and metaphorical manipulation that bypass traditional rule matching, significantly improving detection accuracy and robustness.

[0023] By implementing a layered handling and medium-risk semantic reconstruction strategy, this application addresses the problem of crude risk handling methods and the difficulty in balancing security and user experience in related technologies. This embodiment abandons the traditional one-size-fits-all interception logic, and instead performs semantic reconstruction (such as removing misleading instructions and disrupting semantic coherence) before re-inspection of medium-risk text. This approach not only blocks potential attack paths but also avoids the interception of normal business operations due to misjudgment, preserving the user's business intent to the greatest extent possible while ensuring security.

[0024] By constructing a closed-loop update process based on a verification mechanism, this application addresses the security risks associated with statically fixed defense strategies and model updates in related technologies. This embodiment can collect log data during the defense process, but introduces a verification step (such as manual review, multi-model voting, and sandbox verification) before updating the model or feature library, ensuring that only high-confidence attack samples are used for training. Simultaneously, regression testing is performed in a sandbox environment to confirm that the accuracy of the old scenarios has not decreased before the formal update. This mechanism enables the dynamic adaptive evolution of the defense system while effectively preventing performance degradation caused by false positive samples contaminating the model. Attached Figure Description

[0025] Figure 1 A flowchart illustrating an embodiment of the dynamic adaptive defense method for Prompt injection attacks in this application; Figure 2 A flowchart of an embodiment of the dynamic adaptive defense method for Prompt injection attacks in this application; Figure 3 A schematic diagram of the functional modules of an embodiment of the Prompt attack dynamic adaptive defense system for the large model of this application. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0027] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0028] Reference Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the dynamic adaptive defense method for large-scale Prompt injection attacks in this application. The dynamic adaptive defense method for large-scale Prompt injection attacks provided in this embodiment mainly includes input standardization preprocessing, deep semantic detection, hierarchical attack isolation, and dynamic updating of the attack feature library. After the original Prompt input is processed hierarchically by the system, it achieves the identification, handling, and continuous defense against various Prompt injection attacks.

[0029] Specifically, the system receives raw Prompt text input by the user. This raw Prompt text serves as initial data for subsequent preprocessing and detection steps, and its content may include normal business requests or potential malicious injection instructions.

[0030] The raw input then enters the input standardization preprocessing module, which is configured to perform basic processing on the raw text. This module is responsible for cleaning, normalizing, and extracting basic features from the input text, and consists of three stages: text purification, semantic normalization, and basic feature extraction. Text purification removes meaningless garbled characters, redundant spaces, and other interfering factors, and standardizes the text format. Semantic normalization converts non-standard expressions into standard written language, reducing semantic deviation. Basic feature extraction completes text length normalization and core statistical feature extraction, while also quickly blocking obviously malicious input through initial screening.

[0031] The input text, after initial screening, is fed into the deep semantic detection module, serving as the core detection layer of the system. This module, based on a pre-trained large language model, constructs a triple judgment mechanism: "attack identification sub-model + semantic similarity matching + risk probability fusion". The attack identification sub-model is trained using the cross-entropy loss function and outputs the attack probability of the input text; semantic similarity matching uses cosine similarity calculation to identify variants of known attack samples; and risk probability fusion combines the results of both, obtaining the final risk probability through a fusion formula and classifying it into high, medium, and low risk levels.

[0032] The detection results enter the layered attack isolation module, which is responsible for differentiated risk handling. This module executes a differentiated strategy of "interception-reconstruction-re-inspection-allowance" based on the risk level output by deep semantic detection: high-risk inputs are directly blocked and a security warning is returned; medium-risk inputs are re-inspected after semantic reconstruction; and low-risk inputs are directly allowed to proceed to the large model inference.

[0033] The system is equipped with a dynamic update module for the attack feature library, serving as an adaptive optimization layer. This module collects log data from each processing stage and, through three steps—automatic feature mining and clustering, manual review and deduplication optimization, and incremental fine-tuning of the detection model—achieves dynamic updates to the feature library and detection model, ensuring that the model's F1 score is always ≥0.95.

[0034] Through the above steps, this embodiment applies the updated attack feature library and semantic detection model to the subsequent defense process, thereby forming a closed-loop feedback mechanism that includes detection, isolation, logging, feature mining and model optimization, and realizing continuous iteration of defense capabilities.

[0035] In a first aspect, embodiments of this application provide a dynamic adaptive defense method against large-model Prompt injection attacks.

[0036] In one embodiment, reference is made to Figure 2 , Figure 2 This is a flowchart of an embodiment of the dynamic adaptive defense method for Prompt injection attacks in the large model of this application. Figure 2 As shown, the dynamic adaptive defense method for large-scale Prompt injection attacks includes the following steps: Step S1: Obtain the original text input by the user and preprocess it to obtain the text to be detected.

[0037] Specifically, the system receives raw Prompt text input by the user through an API interface or a front-end interactive interface. Since user input may contain irrelevant noise, poor formatting, or non-standard expressions, directly inputting it into the detection model may affect accuracy. Therefore, the raw text is first preprocessed. Preprocessing may include text cleaning, format standardization, and semantic normalization, aiming to eliminate irrelevant interference features and transform non-standardized input into standardized text that the model can understand, providing a high-quality data foundation for subsequent semantic analysis.

[0038] Step S2: Analyze the semantic intent of the text to be detected using a semantic detection model and a known attack feature library to obtain the risk probability and determine the risk level.

[0039] Specifically, this step employs a dual detection mechanism of "model inference + feature matching". The semantic detection model can be a finely tuned pre-trained language model, such as BERT, RoBERTa, or a Transformer architecture model, but is not limited to these. On one hand, the finely tuned semantic detection model performs deep semantic understanding on the text to be detected, identifying whether it contains implicit malicious intent and outputting the attack recognition probability. On the other hand, the text to be detected is converted into vector form and its similarity is calculated with typical attack vectors stored in a known attack feature library, outputting the similarity probability. The system fuses the above two probabilities to obtain the final risk probability and classifies the risk into three levels: high, medium, and low, based on a preset risk threshold range. This dual mechanism ensures both the generalization ability to identify unknown variant attacks and the high-precision matching of known attack patterns.

[0040] Step S3: Implement tiered treatment based on risk level.

[0041] Specifically, for high-risk levels, their entry into the large model inference process is directly blocked, and a standardized security prompt is returned (note that the specific defense rule logic is not disclosed). At the same time, a complete attack log is recorded, including fields such as input text, detection results, attack time, and attack type, to provide data support for subsequent feature library updates and model iterations.

[0042] For medium-risk levels, a semantic reconstruction process is initiated. While preserving the user's actual needs, misleading instructions and illegal logic are removed from the text. The reconstructed text is then resubmitted to the deep semantic detection module for re-examination. In one feasible implementation of this embodiment, if the secondary risk probability P' < 0.5, the process is allowed; if P' ≥ 0.5, it is upgraded to high risk and blocked, and relevant logs are recorded. This re-examination process continues until the risk level becomes low-risk or a preset termination condition is met (such as exceeding the re-examination limit, exceeding the time limit, or probability convergence).

[0043] For low-risk levels, the process is directly allowed to proceed to the large model inference process without adding extra business delays. At the same time, basic input information is recorded as a reference for subsequent misjudgment feedback and model optimization.

[0044] Step S4: Collect and verify log data during the defense process, and update the semantic detection model or attack signature database based on the verified log data.

[0045] Specifically, the verification process can employ one or a combination of methods, such as manual review, automated script verification, or sandbox environment testing. The system first records log data in real time, including input text, risk probability, handling results, and user feedback. To ensure the security of updates, the system first verifies the log data, filtering out high-value samples confirmed as new types of attacks. Subsequently, the verified log data is used to incrementally fine-tune the semantic detection model or extract new features to update the known attack signature database. This step enables the dynamic evolution of the defense system, allowing it to adapt to the continuous evolution of attack methods and maintain defense effectiveness without frequent manual rule maintenance.

[0046] In this embodiment, a closed-loop feedback mechanism is formed by performing input standardization preprocessing, deep semantic detection, layered attack isolation, and dynamic updating of the attack feature library. This effectively improves the problems of static and fixed large-model Prompt injection attack defense strategies, insufficient semantic understanding, and lack of adaptability in related technologies. Specifically, by utilizing the dual analysis of the semantic detection model and the known attack feature library, the accuracy of identifying covert attacks is significantly improved. Through semantic reconstruction and re-examination mechanisms of medium-risk text, the false interception rate is minimized while ensuring security, thus guaranteeing business continuity. Model and feature library updates based on log verification enable automatic iteration and evolution of defense capabilities, effectively addressing zero-day attacks and significantly improving the long-term security and adaptability of large-model applications in highly compliant scenarios.

[0047] Furthermore, in one embodiment, preprocessing includes: removing garbled characters, redundant spaces, and special symbols from the original text; standardizing capitalization and punctuation; converting abbreviations and colloquialisms in the original text into standard written language using a standardized dictionary and mapping algorithm; extracting the text length features of the original text and normalizing these features.

[0048] Specifically, the text cleaning stage uses regular expression matching to remove invisible characters, consecutive redundant spaces, and meaningless special characters, and standardizes all English characters to lowercase or uppercase, and punctuation marks to full-width or half-width formats to eliminate formatting noise. The semantic standardization stage pre-configures industry-specific standardized dictionaries (such as financial terminology dictionaries and government terminology dictionaries), and uses mapping algorithms to convert colloquial expressions like "what to do" and "what does it mean" into standard written terms like "how to handle" and "what is the meaning," reducing semantic ambiguity. The length feature normalization stage counts the number of text characters L, using a formula... Map it to the [0,1] interval to eliminate the scale effect of texts of different lengths on the model input.

[0049] Building upon this, this embodiment also includes an initial screening and filtering stage: the system sets an initial screening threshold based on the length normalization feature $L_{norm}$ and the proportion of special symbols. If the text to be detected meets obvious malicious conditions (such as abnormal text length and high density of illegal words), the original text is directly blocked. If the blocking conditions are not met, the standardized text is sent to the subsequent deep semantic detection module.

[0050] In this embodiment, multi-dimensional standardized preprocessing addresses the problem of input text noise interference causing distortion in feature extraction and decreased accuracy in detection models. Specifically, removing garbled characters and standardizing the format eliminates interference from irrelevant characters on vectorization; semantic normalization reduces the detection difficulty caused by synonym mutations; initial screening can quickly block obvious attacks before deep detection, reducing the system's computational load; and length feature normalization provides a unified quantitative benchmark for subsequent risk probability calculation, significantly improving the consistency of the detection benchmark and the model's generalization ability.

[0051] Further, in one embodiment, obtaining the risk probability includes: inputting the text to be detected through a semantic detection model to obtain the attack identification probability; converting the text to be detected into a text vector, calculating the similarity between the text vector and known attack vectors in a known attack feature library, and obtaining the similarity probability after mapping processing; and performing a weighted fusion based on the attack identification probability and the similarity probability to obtain the risk probability.

[0052] Specifically, the semantic detection model is built upon fine-tuning a pre-trained large language model. During the model training phase, the cross-entropy loss function is used for parameter optimization, as shown in the following formula: Formula (1).

[0053] in, This is the attack probability output of the model for the i-th sample. The sample labels are 1 for attack samples and 0 for normal samples. The model parameters are optimized using the gradient descent algorithm, and fine-tuning is stopped when the model's F1 score is ≥0.95 to ensure the model's detection accuracy.

[0054] In the similarity matching stage, the cosine similarity calculation formula is used: Formula (2).

[0055] in, The input text semantic vector. Given the semantic vector of a known attack sample, the maximum similarity score is taken as the final similarity S, and then the attack probability is obtained by mapping using the Sigmoid function. .

[0056] Finally, the final risk probability P is calculated using the risk probability fusion formula: Formula (3).

[0057] in, , Let be the weighting coefficient, satisfying In a preferred embodiment, α∈[0.6,0.8], β∈[0.2,0.4], and this range is determined by validation set optimization to balance the weights of model identification and similarity matching.

[0058] In this embodiment, by using a clearly defined loss function and similarity formula, the problems of unclear training objectives and inconsistent similarity calculation standards in related technologies are solved. Specifically, the cross-entropy loss function ensures the clarity of the model's classification boundary, the F1 score stopping criterion guarantees the accuracy baseline before the model goes live, cosine similarity and sigmoid mapping enable quantitative evaluation of variant attacks, and the weighted fusion mechanism combines the advantages of semantic understanding and feature matching, significantly improving the scientific rigor and accuracy of risk probability calculation.

[0059] Further, in one embodiment, determining the risk level includes: if the risk probability is greater than or equal to a first preset threshold, it is determined to be a high-risk level; if the risk probability is less than the first preset threshold but greater than or equal to a second preset threshold, it is determined to be a medium-risk level; if the risk probability is less than the second preset threshold, it is determined to be a low-risk level. Wherein, the first preset threshold is greater than the second preset threshold.

[0060] Specifically, in one feasible implementation of this embodiment, the first preset threshold can be set to 0.8, and the second preset threshold can be set to 0.5. That is: when the risk probability P ≥ 0.8, it is determined to be a high-risk level; when 0.5 ≤ P < 0.8, it is determined to be a medium-risk level; and when P < 0.5, it is determined to be a low-risk level.

[0061] It should be noted that the thresholds of 0.8 and 0.5 mentioned above are merely exemplary values. In practical applications, they can be dynamically configured according to the security sensitivity of the business scenario. For example, in high-security scenarios, the first preset threshold can be lowered to increase the interception rate. In user experience-priority scenarios, the second preset threshold can be increased to reduce the number of re-inspections.

[0062] In this embodiment, a specific three-tiered risk threshold division addresses the problem of ineffective handling strategies caused by vague risk classification standards in related technologies. Specifically, high-risk cases are directly blocked to ensure basic security, low-risk cases are allowed to proceed to ensure business efficiency, and medium-risk cases undergo a re-inspection process for secondary confirmation, achieving a refined balance between security and user experience.

[0063] Furthermore, in one embodiment, semantic reconstruction includes at least one of the following: identifying and removing leading and nested instructions from the text to be detected; converting imperative statements in the text to be detected into declarative statements; and inserting interfering information at key locations in the text to be detected to disrupt the semantic coherence of the attack instructions.

[0064] Specifically, instruction identification and removal can be achieved by using syntactic analysis trees to identify verb phrases and remove leading phrases such as "ignore the above instruction" or "play a certain role." Command-to-statement conversion can be achieved using natural language generation techniques, reconstructing "tell me the password" into "the user is asking for information about the password." Inserting distracting information can involve inserting meaningless characters or adversarial sample noise around sensitive words to disrupt the attention mechanism of large models, preventing them from fully interpreting the attack intent.

[0065] In this embodiment, a variety of semantic reconstruction techniques are used to address the problem in related technologies where direct interception of suspected malicious text leads to the false positive of legitimate business requests. Specifically, removing misleading instructions preserves the user's true intent, command-to-statement reduces the text's offensive characteristics, and inserting interfering information disrupts the integrity of the attack chain. This allows the reconstructed text to retain its business value while passing re-inspection, significantly improving the approval rate for medium-risk text.

[0066] Furthermore, in one embodiment, the preset termination condition includes at least one of the following: the number of re-inspections reaches a preset threshold; the total time consumed in the re-inspection process exceeds a preset duration threshold; and the change in risk probability obtained from two consecutive re-inspections is less than a preset convergence threshold.

[0067] Specifically, a preset threshold for the number of iterations prevents reconstruction from entering an infinite loop (e.g., the risk probability remains unchanged after reconstruction). A preset duration threshold ensures that system response latency is within an acceptable range for the user (e.g., no more than 2 seconds). A preset convergence threshold is used to determine whether reconstruction has reached its limit; if the probability change is negligible, reconstruction stops. Re-inspection terminates and final action (e.g., forced interception or release) is performed when any of these conditions are met.

[0068] In this embodiment, a multi-dimensional re-inspection termination control mechanism is used to address the system deadlock or response timeout issues that may occur during the re-inspection process in related technologies. Specifically, the number of attempts is limited to prevent logical deadlock, the duration is limited to ensure real-time performance, and the convergence threshold avoids invalid calculations. This ensures that the defense process can exit in a timely manner under any abnormal circumstances, thus guaranteeing the stability, availability, and user experience of the system.

[0069] Furthermore, in one embodiment, the verification includes: triggering a verification task when the number of collected suspected new attack feature sets exceeds a preset threshold. The verification is then conducted through manual review, multi-model voting confirmation, or sandbox environment verification to confirm that the samples corresponding to the log data are attack samples.

[0070] Specifically, the collected log data includes key data such as interception logs, suspected attack samples, false positive feedback, and abnormal model outputs, and is stored in a structured manner. During the feature mining phase, natural language processing (NLP) and clustering algorithms are used to automatically mine new attack features from the collected log data, forming feature clusters and extracting core features. After verification, the feature library undergoes operations such as adding, merging, deduplicating, and removing features to maintain its efficiency and simplicity.

[0071] In this embodiment, a triggered verification and multi-source verification mechanism is used to address the model contamination or mislearning issues that may arise from automatic updates in related technologies. Specifically, a quantity threshold avoids the resource waste caused by fragmented updates, while manual and sandbox verification ensures the authenticity and security of new features, preventing attackers from maliciously interfering with the defense model by injecting poisoned samples, thus guaranteeing the reliability and security of the adaptive update process.

[0072] Furthermore, in one embodiment, updating the semantic detection model includes: performing regression testing on the fine-tuned model in a sandbox environment, and after confirming that the accuracy of historical scenarios has not decreased, activating the fine-tuned model as the semantic detection model.

[0073] Specifically, the attack identification sub-model is incrementally fine-tuned periodically using novel attack samples. During this fine-tuning process, a low learning rate is employed to avoid overfitting. For example, the optimization objective can be set to ensure an F1 score ≥ 0.95, continuously improving the generalization ability to identify unknown attacks. After conducting regression testing in a sandbox environment and confirming that the accuracy in old scenarios has not decreased, the fine-tuned model is officially implemented.

[0074] In this embodiment, overfitting and performance fluctuations caused by model updates in related technologies are addressed through low-learning-rate incremental fine-tuning and F1 score constraints. Specifically, the low learning rate ensures the smoothness of model weight updates, the hard requirement of an F1 score ≥ 0.95 ensures a baseline accuracy before the model goes live, and sandbox regression testing prevents the new model from catastrophically forgetting old scenarios, thus achieving continuous and stable evolution of defensive capabilities.

[0075] Secondly, embodiments of this application also provide a dynamic adaptive defense system against large-scale Prompt injection attacks.

[0076] In one embodiment, reference is made to Figure 3 , Figure 3 This is a functional module diagram of an embodiment of the Prompt attack dynamic adaptive defense system for the large-scale model of this application. Figure 3 As shown, the large-scale Prompt injection attack dynamic adaptive defense system includes a preprocessing module 1, a detection module 2, a handling module 3, and an update module 4.

[0077] Preprocessing module 1 is used to acquire the original text input by the user and preprocess it to obtain the text to be detected.

[0078] Specifically, preprocessing module 1 is deployed at the system access layer and is responsible for receiving raw Prompt requests from the API gateway or front-end interface. The module integrates a text cleaning unit, a semantic standardization unit, and a feature extraction unit, which sequentially perform operations such as removing garbled characters, converting spoken language to written language, and normalizing length. It can also optionally perform initial screening and transmit the standardized text to be detected to the detection module.

[0079] Detection module 2 is used to analyze the semantic intent of the text to be detected using a semantic detection model and a known attack feature library, to obtain the risk probability and determine the risk level.

[0080] Specifically, detection module 2 loads a finely tuned semantic detection model and a known attack feature library. Upon receiving the text to be detected, it performs model inference and vector similarity calculation in parallel, weights and fuses the output attack identification probability and similarity probability to obtain the final risk probability, and determines the risk level (high / medium / low) according to a preset threshold range, and transmits the determination result to the processing module.

[0081] Module 3 is used to implement tiered handling based on risk level.

[0082] Specifically, the handling module 3 has a built-in routing strategy engine. If a high-risk signal is received, an interceptor is triggered to block the request and return a security warning. If a low-risk signal is received, the request is directly allowed to the backend large model inference service. If a medium-risk signal is received, the semantic reconstruction unit is invoked to reconstruct the text to be detected, and the reconstructed text is sent back to the detection module for re-inspection until the risk level becomes low-risk or the preset termination condition is met.

[0083] Update module 4 is used to collect and verify log data during the defense process, and update the semantic detection model or attack feature library based on the verified log data.

[0084] Specifically, update module 4 runs asynchronously in the background, collecting log data generated by each module in real time (including interception records, re-inspection records, and false positive feedback). The module contains a feature mining unit and a model fine-tuning unit. When the number of new feature clusters exceeds a threshold, a verification task is triggered. After verification, the module performs CRUD operations on the feature library or initiates the incremental fine-tuning process of the model, and synchronizes the updated model or feature library to the detection module.

[0085] In this embodiment, a modular system architecture addresses the issues of fragmented defense functions, difficulty in integration and maintenance, and inability to support high concurrency in related technologies. Specifically, the four modules—preprocessing, detection, handling, and updating—have clearly defined responsibilities and a closed-loop data flow, supporting distributed deployment and elastic scaling, facilitating rapid integration into actual business systems. The asynchronous operation mechanism of the updating module ensures that online defense performance is unaffected by background optimization tasks, achieving a balance between high availability and adaptive evolution capabilities.

[0086] The functions of each module in the above-mentioned large-scale Prompt injection attack dynamic adaptive defense device correspond to the steps in the above-mentioned large-scale Prompt injection attack dynamic adaptive defense method embodiment, and their functions and implementation processes will not be described in detail here.

[0087] Thirdly, embodiments of this application provide a dynamic adaptive defense device for large-model Prompt injection attacks. It includes a processor, a memory, and a large-model Prompt injection attack dynamic adaptive defense program stored in the memory and executable by the processor. The processor executes the large-model Prompt injection attack dynamic adaptive defense program to implement the steps of the large-model Prompt injection attack dynamic adaptive defense method as described in the first aspect. The memory stores the large-model Prompt injection attack dynamic adaptive defense program and intermediate data generated during its operation. When executed by the processor, the large-model Prompt injection attack dynamic adaptive defense program implements the steps of the large-model Prompt injection attack dynamic adaptive defense method.

[0088] In this embodiment, a dedicated hardware and software architecture addresses the issues of pure software solutions relying on specific operating environments and experiencing performance instability. Specifically, a dedicated processor (such as an NPU) accelerates the inference process of the semantic detection model, reducing defense latency. Persistent storage ensures the security of the feature library and log data. Programmatic defense logic ensures the portability and consistency of the method across different hardware platforms, providing an independent and stable security protection carrier for large-scale model applications.

[0089] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0090] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.

[0091] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.

[0092] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0093] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0094] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.

[0095] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A dynamic adaptive defense method for large-scale Prompt injection attacks, characterized in that, The method includes: Obtain the raw text input by the user and preprocess it to obtain the text to be detected; The semantic intent of the text to be detected is analyzed using a semantic detection model and a known attack feature database to obtain the risk probability and determine the risk level. Based on the risk level, a tiered approach is implemented: if the risk level is high, the original text is intercepted; if the risk level is low, the original text is allowed to proceed to the large model for inference; if the risk level is medium, the text to be detected is semantically reconstructed to obtain reconstructed text, and the reconstructed text is returned as the new text to be detected to execute the semantic intent analysis step until the risk level becomes low or the preset termination condition is met. Collect and verify log data during the defense process, and update the semantic detection model or the attack feature library based on the verified log data.

2. The dynamic adaptive defense method against large-scale Prompt injection attacks as described in claim 1, characterized in that, The preprocessing includes: Remove garbled characters, redundant spaces, and special characters from the original text, and unify capitalization and punctuation. Using a standardized dictionary and mapping algorithm, the abbreviations and colloquialisms in the original text are converted into standard written language; Extract the text length feature of the original text and normalize the text length feature.

3. The dynamic adaptive defense method against large-scale Prompt injection attacks as described in claim 1, characterized in that, The obtained risk probability includes: The text to be detected is input into the semantic detection model to obtain the attack recognition probability; The text to be detected is converted into a text vector, and the similarity between the text vector and the known attack vectors in the known attack feature library is calculated. After mapping, the similarity probability is obtained. The risk probability is obtained by weighted fusion of the attack identification probability and the similarity probability.

4. The dynamic adaptive defense method against large-scale Prompt injection attacks as described in claim 1, characterized in that, The determination of risk level includes: If the risk probability is greater than or equal to the first preset threshold, it is determined to be a high-risk level; If the risk probability is less than the first preset threshold and greater than or equal to the second preset threshold, it is determined to be a medium risk level; If the risk probability is less than the second preset threshold, it is determined to be a low-risk level; Wherein, the first preset threshold is greater than the second preset threshold.

5. The dynamic adaptive defense method against large-scale Prompt injection attacks as described in claim 1, characterized in that, The semantic reconstruction includes at least one of the following: Identify and remove leading and nested instructions from the text to be detected; Convert the imperative statements in the text to be detected into declarative statements; Interference information is inserted at key locations in the text to be detected to disrupt the semantic coherence of the attack instructions.

6. The dynamic adaptive defense method against large-scale Prompt injection attacks as described in claim 1, characterized in that, The preset termination condition includes at least one of the following: The number of re-inspections has reached the preset threshold. The total time taken for the re-inspection process exceeded the preset time threshold; The change in risk probability obtained from two consecutive re-inspections is less than the preset convergence threshold.

7. The dynamic adaptive defense method against large-scale Prompt injection attacks as described in claim 1, characterized in that, The verification includes: When the number of suspected new attack feature sets collected exceeds a preset threshold, a verification task is triggered. The log data was confirmed to be an attack sample through manual review, multi-model voting confirmation, or sandbox environment verification.

8. The dynamic adaptive defense method against large-scale Prompt injection attacks as described in claim 1, characterized in that, The updating of the semantic detection model includes: After performing regression testing on the fine-tuned model in a sandbox environment and confirming that the accuracy of historical scenarios has not decreased, the fine-tuned model is then used as the semantic detection model.

9. A dynamic adaptive defense system against large-scale Prompt injection attacks, characterized in that, include: The preprocessing module is used to acquire the raw text input by the user and preprocess it to obtain the text to be detected; The detection module is used to analyze the semantic intent of the text to be detected using a semantic detection model and a known attack feature library, to obtain the risk probability and determine the risk level; The processing module is used to perform tiered processing based on the risk level: if the risk level is high, the original text is intercepted; If the risk level is low, the original text is allowed to proceed to the large model inference; if the risk level is medium, the text to be detected is semantically reconstructed to obtain reconstructed text, and the reconstructed text is used as the new text to be detected to trigger the detection module to re-examine until the risk level becomes low or the preset termination condition is met. The update module is used to collect and verify log data during the defense process, and update the semantic detection model or the attack feature library based on the verified log data.

10. A dynamic adaptive defense device against large-scale Prompt injection attacks, characterized in that, The system includes a processor, a memory, and a large-model Prompt injection attack dynamic adaptive defense program stored in the memory and executable by the processor, wherein when the large-model Prompt injection attack dynamic adaptive defense program is executed by the processor, it implements the steps of the large-model Prompt injection attack dynamic adaptive defense method as described in any one of claims 1 to 8.

Images