A multi-layer defense mechanism for large language model security and an implementation system thereof

Abstract

This invention discloses a multi-layered defense mechanism and its implementation system for the security of large language models. The mechanism includes: at the current time step, based on the generated tokens and received user input text, generating an original probability distribution vector using the large language model; calling a rule engine to perform hard constraint processing on the original probability distribution vector, generating a candidate token containing at least one candidate token; if the triggering policy is met, using a pre-built risk model to assess the risk of the candidate token and obtain a risk score; determining whether the candidate token meets the constraints and whether the risk score is less than a dynamic threshold; otherwise, executing a blocking policy; if yes, outputting the candidate token; updating the guard state at the current time step based on the judgment result and proceeding to the next time step, until the termination condition is met, decoding all generated tokens into response text and outputting it. Compared to schemes that only post-process the generation results, this invention can intervene in the generation process earlier and with greater precision.

Description

Technical Field

[0001] This invention relates to the field of model defense technology, specifically to a multi-layered defense mechanism and its implementation system for the security of large language models. Background Technology

[0002] In recent years, large language models have been widely applied in various fields such as intelligent question answering, dialogue systems, content creation, and decision support due to their superior performance in natural language understanding and generation. However, because these models typically employ autoregressive generation mechanisms based on probability distributions, they are easily influenced by user input in open scenarios, resulting in output content that does not comply with security specifications or business constraints. Therefore, generation security protection technologies for large language models have become one of the key research focuses in related fields.

[0003] Existing security measures for large language models mainly focus on two stages: pre-generation and post-generation. In the pre-generation stage, system prompts or preset rules are typically used to guide the model's behavior; in the post-generation stage, methods such as keyword filtering, rule matching, or content review are used to intercept or process the generated text.

[0004] However, the aforementioned solutions are mostly static constraints or post-hoc protections, making it difficult to effectively control semantic evolution during model generation. In practical applications, the model may gradually drift towards a high-risk semantic trajectory during multiple generation rounds, and existing solutions typically lack the ability to perceive and intervene in this process in real time. Furthermore, the lack of continuous recording and comprehensive evaluation of risk states throughout the historical generation process makes it difficult to dynamically adjust protection strategies based on changes in risk over time, thus limiting the granularity of generation security control.

[0005] Therefore, how to effectively constrain the output of large language models during the generation process and achieve dynamic and continuous risk control by combining historical generation states remains a technical problem to be solved in this field. Summary of the Invention

[0006] To address the problem that existing solutions, which mainly focus on the pre-generation and post-generation stages, cannot meet the requirements of both generation security and output quality in complex interactive scenarios, this invention provides a multi-layered defense mechanism and its implementation system for the security of large language models.

[0007] To achieve the above objectives, the technical solution of the present invention is as follows:

[0008] Firstly, this application discloses a multi-layered defense mechanism for the security of large language models, comprising the following steps:

[0009] At the current time step, based on the generated token and the received user input text, the original probability distribution vector is generated using a large language model;

[0010] The rule engine is invoked to perform hard constraint processing on the original probability distribution vector, generate a rule mask, and renormalize the original probability distribution vector according to the rule mask. The renormalization result is then used to generate a probability selection that includes at least one candidate token. The rules executed by the rule engine include token blacklist, phrase completion blocking, and structure trigger blocking.

[0011] The triggering strategy for semantic checks is determined based on the guarding state of the previous time step. If the triggering strategy is met, the candidate tokens are assessed for risk using a pre-built risk model to obtain a risk score. The guarding state includes at least the cumulative risk score and operating mode that represent the overall risk level over the time dimension.

[0012] Determine whether the candidate token meets the rule constraints and whether the risk score is less than the dynamic threshold; otherwise, execute the blocking strategy; if yes, output the candidate token; the blocking strategy includes resampling, track correction and suspension of response rejection;

[0013] Update the guard state at the current time step based on the judgment result and proceed to the next time step until the termination condition is met, and decode all generated tokens into reply text and output them.

[0014] Secondly, this application introduces an implementation system for a multi-layered defense mechanism for the security of large language models, including a probability estimation module, a candidate generation module, a risk assessment module, a blocking strategy decision module, and a result output module.

[0015] The probability estimation module is used to generate the original probability distribution vector at the current time step, based on the generated token and the received user input text, using a large language model.

[0016] The candidate generation module calls the rule engine to perform hard constraint processing on the original probability distribution vector, generate a rule mask, and renormalize the original probability distribution vector according to the rule mask. The renormalization result is then used to perform probability selection to generate a token containing at least one candidate token. The rules executed by the rule engine include token blacklist, phrase completion blocking, and structure trigger blocking.

[0017] The risk assessment module is used to determine the triggering strategy for semantic checks based on the guarding state of the previous time step. If the triggering strategy is met, the candidate token is assessed for risk using a pre-built risk model to obtain a risk score. The guarding state includes at least the cumulative risk score and the operating mode that represent the overall risk level in the time dimension.

[0018] The blocking strategy decision module is used to determine whether the candidate token meets the rule constraints and whether the risk score is less than the dynamic threshold; otherwise, the blocking strategy is executed, and if yes, the candidate token is output; the blocking strategy includes resampling, track correction and suspension of rejection;

[0019] The result output module is used to update the guard state at the current time step based on the judgment result and proceed to the next time step until the termination condition is met, and decode all generated tokens into reply text for output.

[0020] Thirdly, this application introduces a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the multi-layered defense mechanism for the security of large language models as described above.

[0021] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0022] 1. This application directly applies rule constraints to the original probability distribution vector output by the large language model at each time step, so that the security rules directly act on the token generation probability space, thereby reducing the possibility of violating tokens being selected from the source. Compared with the scheme that only processes the generation results, it can intervene in the generation process earlier and more precisely.

[0023] 2. By maintaining the cumulative risk score and operating mode in the guard state, this application can continuously perceive and record the changing trend of risk during the multi-time step generation process, avoid making isolated judgments based on a single output, help identify implicit and progressive high-risk generation tracks, reduce the accumulation of generation risks in the time dimension, and improve the security, stability and overall availability of the generated content. Attached Figure Description

[0024] The disclosure of this invention is illustrated with reference to the accompanying drawings. It should be understood that the drawings are for illustrative purposes only and are not intended to limit the scope of protection of this invention. In the drawings, the same reference numerals are used to refer to the same parts. Wherein:

[0025] Figure 1 This is a flowchart illustrating a multi-layered defense mechanism for the security of large language models, as described in this invention.

[0026] Figure 2 For based on Figure 1 The logical flowchart of the execution blocking strategy;

[0027] Figure 3 For based on Figure 1 The leakage window ratio indicator chart;

[0028] Figure 4 For based on Figure 1 The chart showing the success rate of course correction;

[0029] Figure 5 For based on Figure 1 A graph showing the accidental injury rate.

[0030] Figure 6 This is an architecture diagram of the implementation system for the multi-layered defense mechanism for the security of large language models, as described in Example 2.

[0031] Figure 7 For based on Figure 6 The execution logic flowchart of the implementation system. Detailed Implementation

[0032] It is readily understood that, based on the technical solution of this invention, those skilled in the art can propose various interchangeable structural methods and implementations without altering the essential spirit of the invention. Therefore, the following detailed embodiments and accompanying drawings are merely illustrative examples of the technical solution of this invention and should not be considered as the entirety of the invention or as limitations or restrictions on the technical solution of this invention.

[0033] In existing technologies, the security protection of large language models typically relies on pre-generation prompts and constraints or post-generation content filtering. Security assessments are mostly based on single output results or static rules, lacking a unified data coordination mechanism between rule information, probability generation results, and semantic risk assessment. Because the output probability distribution of the model evolves continuously with the context during multi-timestep generation, existing solutions struggle to promptly perceive the changing trends of potential risk tokens in the probability space, and cannot continuously feed back risk information exposed during historical generation into subsequent generation decisions. This results in a disconnect between security control and the generation process, a lack of temporal continuity in risk assessment, and a tendency for semantic shifts to occur gradually in long contexts or adversarial input scenarios without being effectively intercepted.

[0034] To address the aforementioned issues, this application introduces a multi-layered defense mechanism in the large language model generation process, linking probability distribution-based rule constraints, risk assessment, and guard states. This mechanism integrates rule masks, candidate token risk scores, and historical cumulative risk states into a unified generation decision framework, ensuring continuous correlation between generated data at different time steps. By directly constraining and renormalizing the probability distribution through rule masks, combined with the dynamic guidance of guard states for semantic check triggering and threshold adjustment, risk assessment results can continuously feed back and influence subsequent generation paths. Furthermore, when necessary, resampling, trajectory correction, or generation termination enable proactive control of the generation trajectory. This constructs a security protection mechanism for the co-evolution of rule constraints and semantic risks over time, effectively improving the overall security and controllability of the large language model generation process.

[0035] After introducing the basic concept of the present invention, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0036] Example 1

[0037] like Figure 1 and Figure 2 The diagram illustrates a multi-layered defense mechanism for the security of large language models, comprising the following steps:

[0038] S100. At the current time step, based on the generated token and the received user input text, generate the original probability distribution vector using a large language model.

[0039] Before receiving user input text and starting text generation, the rule engine is configured and the guard mode is initialized. The specific steps are as follows:

[0040] The scenario is determined based on the user's input text, and the rule configuration that matches the scenario is retrieved from the preset configuration library and constructed into a rule engine.

[0041] Set the cumulative risk score to the preset minimum risk value and the operating mode to normal mode to complete the guard state initialization.

[0042] Specifically, after receiving the user's input text, the context object for this generated session is constructed, which includes at least: the user input text x, the session identifier (such as session_id), and a prefix cache for subsequent token-by-token generation. (That is, an ordered sequence of all tokens that have been confirmed and output by the system before the current generation time step t). When generation has not yet started, t can be set to 1 and the prefix buffer can be an empty sequence, so that the distribution of the next token can be directly inferred based on x in the first time step. This context object serves as a shared carrier for subsequent rule matching, risk assessment, and logging, ensuring that different modules use consistent input and state under the same session dimension.

[0043] Perform scene determination on the user input text x to obtain the scene identifier scene_id for strategy selection. Scene determination can be performed using any of the following methods or a combination thereof:

[0044] 1) Rule-based judgment based on keywords / regular expressions / templates (e.g., detecting whether it contains a specific domain thesaurus, sensitive intent patterns, format structure trigger symbols, etc.);

[0045] 2) Intent recognition based on lightweight classification models (e.g., outputting categories such as "general dialogue / code generation / medical consultation / financial consultation / jailbreak inducement");

[0046] 3) Determine the scenario based on system-side business metadata (e.g., product function entry points, user permission levels, tenant policies, etc.).

[0047] The output scene identifier scene_id will be used as the index key for retrieving the configuration library, ensuring that the subsequently loaded rule sets, thresholds and action mappings are consistent with the current input scene.

[0048] Based on the scene identifier `scene_id`, the system retrieves rule configurations matching the scene from a pre-defined configuration library. These rule configurations include at least one or more of the following: a token blacklist set, phrase completion blocking rules, and structure trigger blocking rules, along with corresponding triggering conditions, severity, and action mapping parameters. The retrieved rule configurations are then compiled / constructed into an executable rule engine, `RuleEngine`, which may include:

[0049] Map the token blacklist to a set of vocabulary indexes or a Boolean lookup table structure (for quickly generating masks at the vocabulary level).

[0050] Compile phrase completion masking rules into regular expressions, DFA state machines, or prefix tree structures (for use with prefix buffers). (Perform prefix matching and deduce the set of prohibited tokens);

[0051] Compile the structure trigger masking rules into a format pattern detector (e.g., triggers for structure signals such as “step instructions”, “code block structures”, and “recipe / dosage / parameter tables”) to support the imposition of hard constraints on candidate tokens when a specific context structure occurs.

[0052] This step outputs an executable rule engine, RuleEngine (and its internal compilation artifacts), enabling it to be used in every subsequent time step based on the same session context x. Generate rule mask .

[0053] Initialize the guard state before generation begins. This is used to represent risk trends and control strategies over a time dimension. Initialization includes at least:

[0054] Accumulated risk score Set to a preset minimum risk value (e.g., 0 or a lower bound close to 0) to indicate that the current session has not yet been exposed to risk;

[0055] Run mode Set to normal mode;

[0056] Initialize cooling count =0, and recent trigger records This is an empty set, used for subsequent dynamic adjustment of the inspection frequency and action intensity.

[0057] This guard state will be tied to the subsequent generation process and updated at each time step. This enables the system to "remember historical risks and influence subsequent judgments".

[0058] Upon entering the first or any generation time step t, based on the user input text x and the already generated prefix... Perform a forward inference operation on the large language model to obtain the logits vector in the vocabulary dimension. , where V represents the vocabulary size. Then... Perform a softmax operation to obtain the original probability distribution vector. The calculation formula is as follows:

[0059] ;

[0060] in, This represents the probability that the i-th token is selected at time step t. Represents the logits vector The i-th component; Represents the natural exponential function; This indicates that the logits vectors corresponding to all V tokens in the vocabulary are summed for normalization.

[0061] Include the scene identifier (scene_id), the rule engine version (RuleEngine version), and the initial state of the guard. The initialization information, including the current time step 't', is associated with and stored in the shared context object, enabling subsequent steps to directly call the rule engine and read the guard status without repeated parsing. Furthermore, the initialization information can be written to a structured log (e.g., policy version, summary of loaded rule sets) to support subsequent replay and policy regression verification; this log does not affect the main generation process but enhances system maintainability and traceability.

[0062] This step involves scene recognition of the user-input text x and accordingly configuring the rule engine and initializing the guard state, ensuring that the subsequent generation process has a security constraint foundation that matches the current application scenario. This allows the large language model to operate in a controlled environment from the moment it enters token-by-token inference, avoiding uncertainties caused by temporary policy loading or inconsistent states during the generation process. Simultaneously, this step provides a stable data starting point for subsequent dynamic semantic checks based on guard states, risk accumulation updates, and blocking strategy selection. This creates a continuous and adjustable security control loop in the time dimension of the entire generation process, thereby improving the overall security, consistency, and controllability of the system.

[0063] S200. Call the rule engine to perform hard constraint processing on the original probability distribution vector, generate a rule mask, and renormalize the original probability distribution vector according to the rule mask. Perform probability selection on the renormalization result to generate a token containing at least one candidate token. The rules executed by the rule engine include token blacklist, phrase completion blocking, and structure trigger blocking.

[0064] At the current time step t, this step obtains the original probability distribution vector output by the large language model. Prefixes already generated The set of rules (containing at least one or more of the following: token blacklist rules, phrase completion blocking rules, and structure trigger blocking rules). Where V represents the vocabulary size.

[0065] Initialize rule mask This is a vector of all ones, indicating that all tokens are allowed in the initial state. Simultaneously, the set of prohibited tokens is initialized. It is used to aggregate the prohibited token results generated by different rules to ensure that multiple rules can form consistent hard constraints on the same vocabulary space at the same time step.

[0066] Invoke the token blacklist detector, based on a preset blacklist word list or token index set. Directly generate a subset of prohibited tokens and merge it into the set of prohibited tokens. The corresponding position of the mask is set to 0. This process directly hard-blocks explicitly prohibited tokens, ensuring that such tokens cannot be selected as outputs at this time step.

[0067] For example, in certain public-facing dialogue services or industry compliance scenarios, the system needs to prohibit the output of specific sensitive words, abusive words, or regulated terms. In this case, the rule engine can pre-configure a blacklist of tokens corresponding to the word index. When the model outputs the original probability distribution at the current time step, the rule engine directly marks the tokens corresponding to the blacklist as prohibited from generation and sets the corresponding position to 0 in the rule mask, so that the probability of these tokens is always 0 in the renormalized probability distribution, thereby preventing the model from outputting explicitly prohibited words during the generation process.

[0068] The phrase completion masking detector is invoked based on the generated prefix. Determine if the current prefix matches the prefix pattern or state machine state of a certain "restricted phrase"; when a trend of completing a restricted phrase is detected, calculate the set of candidate tokens that will trigger phrase completion. And incorporate it into the prohibited token set. Then, a consistent hard mask is applied to the mask. It can be derived from regular expressions, DFA state machines, prefix trees, or n-gram rules, and is used to block the generation path of "completing a restricted phrase" at the token level.

[0069] For example, in multi-turn dialogues, users might use a "segmented guidance" approach to gradually guide the model to generate certain non-compliant expressions. That is, the outputs in the first few steps are not inherently non-compliant, but if generation continues, they will form restricted phrases or fixed expressions. For this type of scenario, the rule engine can determine whether the current output is close to the prefix form of a restricted phrase based on the generated prefixes, and identify a candidate set of tokens that "choosing certain tokens will cause the phrase to be completed." Subsequently, these tokens that will trigger phrase completion are hard-masked in the rule mask, so that even if the model probabilistically tends to continue completing, it will turn to other expression paths due to the mask constraint, thereby blocking the formation of non-compliant phrases that are "about to be completed" during the generation process.

[0070] The call structure triggers the masking detector based on the user input x and the generated prefix. Determine if a preset structure signal (e.g., numbered step structure, recipe / parameter list structure, code block structure, instructional mode, etc.) has occurred. When the structure triggering condition is met, determine the set of prohibited tokens corresponding to the structure trigger. And incorporate it into the prohibited token set. Then, the corresponding position in the mask is set to 0. Through this process, when a specific generated structure is detected to be forming, hard constraints can be applied to tokens that may further expand the structure, thereby achieving structure-level generation control.

[0071] For example, in some higher-risk interaction scenarios, the prohibited content does not rely on a fixed word, but is often presented in a specific structure, such as organizing the output in the form of "step-by-step instructions," "list-style enumeration," or "code blocks / instruction blocks." In response, the rule engine can perform structure recognition on the user input and the generated prefixes. When it detects that the output is entering a certain restricted structure (such as the appearance of consecutive numbers, specific delimiter patterns, or block format markers), it triggers a structure-based masking rule. This adds the set of tokens that may drive the structure to continue to expand to the prohibited list and sets the corresponding value to zero in the rule mask. This makes it difficult for the model to continue generating along this structure in the subsequent renormalization and sampling stages, thereby achieving pre-constraint on high-risk expression forms.

[0072] When multiple rules are in effect simultaneously, a unified "union masking" method is used to aggregate the mask, meaning that the token is prohibited as long as it triggers any hard rule: . It is the i-th component in the regular mask vector, and its value is 0 or 1.

[0073] To avoid the anomaly of all tokens being banned and unable to be generated, a safety net strategy can be set. If... If necessary, some low-severity rules can be relaxed or a set of fallback rules can be introduced to ensure that subsequent candidate generation still has feasible solutions. For example, outputting EOS / rejection template tokens can be allowed first (safe convergence); only "experience / format-based low-risk rules" can be relaxed, while hard prohibition rules such as blacklist / phrase completion are strictly prohibited from being relaxed.

[0074] The original probability distribution vector Element-by-element hard constraints are applied using a mask to obtain a zero-based distribution vector. The calculation formula is: ;

[0075] in, This represents the median probability value of the i-th token in the zero-distribution vector; Represents the original probability distribution vector The probability value corresponding to the i-th token.

[0076] Then to Performing renormalization yields the corrected distribution vector. The calculation formula is:

[0077] ;

[0078] in, This represents the final probability value of the i-th token in the corrected distribution vector; This represents the summation of the zero-probability values ​​of all V tokens in the vocabulary, used for renormalization.

[0079] in, As the sole input probability distribution for generating subsequent candidate tokens, it ensures that the probability of a token prohibited by the mask is 0, and that the probability of a allowed token remains in a canonical distribution form where the sum of probabilities is 1 within the same vocabulary space.

[0080] The renormalized distribution is based on candidate generation strategy parameters (such as TopK, TopP, temperature, etc.). Perform probabilistic selection to generate a candidate set containing at least one candidate token. .For example:

[0081] TopK method: From Select the K tokens with the highest probability from the list as candidates;

[0082] Sampling method: according to The probability distribution is sampled once or multiple times to obtain at least one candidate token. If necessary, "no repeated sampling" or "number of retries" can be set to improve candidate diversity. Output candidate set. and its corresponding probability information, and , and This information is provided for use in subsequent semantic risk assessment and adjudication steps.

[0083] This step embeds rule constraints pre-processed into the probabilistic generation stage of the large language model. This shifts security control from post-output interception or soft guidance based on prompts to directly applying to the original probability distribution vector at the current time step. The rule mask generated by the rule engine hard-masks illegal or unwanted tokens, reducing their likelihood of being selected in the sampling process from the outset. Especially when configuring rules such as token blacklists, phrase completion masking, and structure trigger masking simultaneously, the rule mask provides a unified constraint expression at the lexical, phrase, and structural levels. This prevents the model from bypassing static filtering through synonym substitution, step-by-step induction, or format reorganization during multi-time-step generation, thereby improving the compliance and controllability of the generation process.

[0084] S300. Determine the triggering strategy for semantic checks based on the guarding state of the previous time step. If the triggering strategy is met, use the pre-built risk model to assess the risk of the candidate token and obtain a risk score. The guarding state includes at least the cumulative risk score and the operating mode that represent the overall risk level in the time dimension.

[0085] Specifically, the strategy for determining the semantic check trigger based on the guard state of the previous time step includes:

[0086] Get the operating mode of the guard state at the previous time step, and determine whether the operating mode is the normal mode;

[0087] If yes, semantic checks are triggered according to the preset compensation interval; otherwise, semantic checks are performed step by step.

[0088] The guard mode operates by comparing the accumulated risk score with a preset threshold range. If the score is below the lower limit of the threshold range, it is in normal mode; if it is within the threshold range, it is in alert mode; and if it is above the upper limit of the threshold range, it is in locked mode.

[0089] Specifically:

[0090] Before performing semantic checks on candidate tokens at the current time step t, obtain the guard state from the previous time step. and extract at least the cumulative risk score. Operating mode Simultaneously read the cooling count (e.g.) ) or recently triggered records (such as This is used to compensate for or tighten the triggering strategy. If there is a candidate set consisting of multiple candidate tokens in S200, all candidate tokens in the candidate set are first input into the risk model for risk assessment. In candidate S400, they are executed according to the risk score from low to high priority. It is determined whether the rule constraints and risk thresholds are met in turn; the first one that meets the requirements is output; if none of them meet the requirements, the blocking strategy is executed.

[0091] According to the operating mode Determine whether a semantic check is triggered at this time step:

[0092] Determine the operating mode Is it in normal mode? If so... If the condition is "normal", then semantic checks are triggered according to a preset compensation interval, meaning the risk model is invoked only when the condition "reaching the compensation interval" is met. The compensation interval can be implemented using a counter or a step interval M. In normal mode, semantic checks can be triggered according to a preset compensation interval, which can be set to trigger once every M tokens, where M can be an integer from 2 to 20. For example, when... =normal and When in the lower range, the compensation interval M can be set to 12; when Approaching the lower limit of the threshold range At this time, M can be reduced to 8 to increase the frequency of semantic checks.

[0093] like If the condition is not "normal" (i.e., "cautious" or "locked"), then a semantic check is triggered step-by-step. This means that at each time step, the risk model is invoked to perform a risk assessment on the candidate token. The trigger identifier for the current time step is then output. ,in =1 indicates that a semantic check is triggered. =0 indicates that semantic checks are skipped.

[0094] when When =1, for the candidate set Construct candidate prefixes from at least one candidate token. Where c represents a single candidate token selected from the candidate set at the current time step t; This represents a sequence concatenation operation, used to append candidate tokens to the end of an existing token prefix sequence to form a new sequence.

[0095] And the user input text x with candidate prefixes Input the pre-built risk model RiskModel to obtain the risk score of the candidate token. Among them, risk score The risk score is used to characterize the risk level of a candidate token in the current context and can be limited to the numerical range [0,1]. If the candidate set contains multiple candidate tokens, the risk score of each candidate can be calculated separately to form a risk score set.

[0096] when When the risk score is 0, to maintain consistency in subsequent state update chains, the risk score for this time step can be set to a default value or an inherited strategy value. For example, the risk score for the current time step can be set to a preset low-risk value; or the risk score for the current time step can be set to the maximum value between the risk score evaluated in the previous round and the preset conservative lower bound; or only the "unevaluated" flag can be recorded, and the weights can be adjusted in subsequent fusion updates. Appropriate compensations are made (e.g., zeroing or reducing) to obtain the representative risk score input for state fusion update at this time step. Available values.

[0097] Specifically, based on the risk assessment results obtained at this time step, a representative risk score is determined for cumulative risk fusion. In one implementation, Available candidate set The maximum risk value. In another implementation, Alternatively, the risk score of the final submitted / hit candidate token can be taken to ensure that the cumulative risk is consistent with the actual generation trajectory.

[0098] Weight It can be associated with assertion severity, trigger type, or operating mode, such as giving higher weight in locked / alert mode and lower weight in normal mode; it can also be related to the risk category triggered in the candidate to achieve differentiated accumulation of different risk types.

[0099] Accumulated risk score from the previous time step Representative risk score at this time step The cumulative risk score for the current time step is obtained by performing a fusion update using the following formula. :

[0100] ;

[0101] The output of this fusion update The cumulative result is used to characterize the overall risk level over time and serves as the basis for subsequent operation mode determination and threshold adaptive control.

[0102] The updated cumulative risk score Compare with a preset threshold range to determine the current time step's operating mode. Specifically, let the lower limit of the threshold interval be... The upper limit is :

[0103] like < Then determine =normal (normal mode);

[0104] like Then determine =cautious (alert mode);

[0105] like > Then determine =locked (locked mode).

[0106] This leads to the updated operating mode. This will be used to select the semantic inspection trigger strategy for the next time step. and Write the current time step guard state It can optionally update fields such as cooldown count and recent trigger records to form a state object that can be read by the next time step. Then... The output is used by the subsequent adjudication and blocking strategy modules, and serves as the basis for entering the next time step. The source of.

[0107] In an optional implementation, the cumulative risk score The value range is [0,1], and the lower limit of the threshold interval is... Available value: 0.30, maximum: A value of 0.70 can be used; however, in other implementations, the threshold range can also be adjusted according to the security strategy of different scenarios.

[0108] Regarding the pre-built risk model, the construction process of the risk model is as follows:

[0109] First, we construct the training data and organize the samples:

[0110] Historical dialogues and generated logs (including prompts, context, model outputs, and human or system review results); rule engine triggered samples (such as phrase completion / structure-triggered fragments); adversarial prompt generated samples (using various prompt templates to induce the model to produce potentially high-risk continuations, used to enhance robustness). Each sample is organized as follows: The input text to the model is constructed into a concatenated sequence, with corresponding labels that can be binary (safe / risky) or multi-class (jailbreak, violation of commands, privacy breach, hate speech, etc.). Sample labeling is primarily done manually (high confidence), supplemented by rule / review system-assisted labeling (for expansion, but with lower weights). Training / validation / testing ratios are 80% / 10% / 10%, ensuring coverage of each risk type in the validation / test sets.

[0111] In one alternative implementation, the RiskModel employs a lightweight Transformer encoder for text discrimination, such as a small model like DistilBERT / MiniLM / DeBERTa-v3-small. The classification head consists of a linear layer (hidden_size → num_labels) with dropout. The input length is 256, and the dropout value is 0.1. Taking binary classification as an example, the output is a risk score. ∈[0,1].

[0112] Secondly, regarding the training objective and loss function, the weighted cross-entropy loss is used for binary classification. For the training hyperparameters, the optimizer is AdamW, with a learning rate of 2e-5, weight decay of 0.01, batch size of 64 (128 if GPU resources are sufficient), training epochs of 3-5, warmup of 0.06, learning rate scheduling of linear decay, and mixed precision of FP16. Validation metrics can include AUC, F1 score, and recall, used to select the optimal checkpoint.

[0113] After training, the validation set is thresholded to obtain the basic threshold. If the output is a binary classification probability, the basic threshold can be 0.5. If it is more conservative, it can be 0.35 to 0.45. The threshold and the mapping of risk type, severity weight, etc. are solidified into the "model configuration file" (for loading at runtime).

[0114] In practical applications, the RiskModel can be initialized before user input is received and before generation begins. Specifically, the scene identifier `scene_id` is read, and the RiskModel configuration matching `scene_id` is loaded from the configuration library. This includes checkpoint path / version number, risk type set and label mapping, base threshold, severity weight mapping, and the RiskModel weights are loaded into memory / GPU (or CPU if no GPU is available). The tokenizer corresponding to the RiskModel is loaded (consistent with training). The input template is initialized by preprocessing the user input `x` (whitespace, full-width / half-width characters, truncated to maximum length, etc.), and the risk model call counter and compensation interval parameters are initialized. When the guard state is initially normal, the semantic check compensation interval `N` is set, and the guard state is... Bind to the triggering strategy of RiskModel. To avoid delays or exceptions on the first call, a lightweight warm-up can be performed before generation: construct a shortest input (e.g., containing only prompt), perform one forward inference; verify that the output dimension and score range are correct; if using GPU / FP16, ensure that the tensor type is consistent with the device.

[0115] In one example scenario, the system is used for a public conversational question-and-answer service. When a user asks a general question at the beginning of the conversation, the accumulated risk score of the guarded state is calculated. At a low level, corresponding to the normal mode, the system triggers semantic checks according to a preset compensation interval. This means the risk model isn't invoked at every time step, but rather a risk score is obtained by assessing candidate tokens when the interval condition is met, thus maintaining the continuity of the generation process in low-risk contexts. As the dialogue continues, if the user input gradually becomes suggestive or the semantic direction shifts, the risk model, when triggered, will output a higher risk score for the candidate token. The system then merges this risk score with the cumulative risk score from the previous time step to obtain a new cumulative risk score. The system compares the results with a threshold range to switch the operating mode from normal mode to alert mode. In alert mode, the system triggers semantic checks step-by-step, allowing the risk model to evaluate candidate tokens at each time step. This enables a more frequent check to capture potentially high-risk continuations during periods of rising semantic risk. If the accumulated risk further increases and exceeds the upper limit of the threshold range, the operating mode enters a locked mode. The system continues to perform semantic checks incrementally and outputs more stringent risk assessment results, providing a stable risk basis for subsequent adjudication and blocking strategies.

[0116] This step, without altering the main model generation mechanism, achieves a closed-loop linkage between risk assessment and state memory, providing a continuous and consistent risk measurement basis for subsequent dynamic threshold adjudication and blocking strategies, thereby improving the overall security controllability and strategy consistency in the generation process.

[0117] S400. Determine whether the candidate token meets the rule constraints and whether the risk score is less than the dynamic threshold. If not, execute the blocking strategy. If yes, output the candidate token. The blocking strategy includes resampling, track correction and suspension of response rejection.

[0118] The specific steps for implementing the blocking strategy are as follows:

[0119] Determine whether the candidate token meets the rule constraints; otherwise, discard the candidate token.

[0120] If so, determine whether the risk score of the candidate token is less than the dynamic threshold of the current time step; if so, output the candidate token.

[0121] Otherwise, the risk level of the candidate token is matched based on the pre-defined mapping table of risk scores and risk levels, and the operating mode of the previous time step and the scenario of the user input text are quantified into risk levels.

[0122] The highest risk level is taken as the severity of the candidate token, and corresponding resampling, track correction, or suspension of response is performed.

[0123] Specifically, at the current time step t, the data required for this step includes the candidate token c and the rule mask value corresponding to that candidate. The candidate's risk score Previous time step operation mode The scene identifier (scene_id) corresponding to the user input text x and its scene risk level, as well as the dynamic threshold of the current time step. .

[0124] Determine if the candidate token meets the rule constraints (compliance):

[0125] Candidate tokens are decoded into text fragments, and lightweight normalization is performed to obtain standard text. The standard text is then checked class by class according to "rule priority" (failure occurs if any rule is matched):

[0126] First, execute the consecutive number rule (numerical strings such as mobile phone number / ID card number);

[0127] Then execute the format confidentiality rule (for multi-variant formats, use DFA / mode state);

[0128] Then execute the cross-token combination rules (combining cross-boundary and sliding window rules).

[0129] Specifically, lightweight standardization includes at least one of the following:

[0130] Number / letter standardization (e.g., converting full-width characters to half-width characters, unifying uppercase and lowercase); separator standardization (merging spaces, hyphens, underscores, etc., into uniform placeholders); removal of redundant symbols that do not affect semantics.

[0131] The rule aims to prevent the output of consecutive numbers up to 11 digits. First, the old state is read, and the standard text is scanned character by character. When a numeric character is detected in the standard text, the consecutive number count is incremented. When a non-numeric character is detected, the consecutive number count is maintained or cleared according to a preset standard (e.g., whether a separator breaks consecutive numbers is determined by the rule). If the consecutive number count reaches or exceeds a preset threshold (e.g., reaching 11 consecutive digits), the candidate token is determined to violate the rule, and the detection ends.

[0132] The goal of the rule is to prohibit the output of variations of a certain "confidential format". For each format / pattern rule, the state machine state corresponding to that rule (such as the current state of the DFA / NFA) is read. The state transition is performed character by character in the standard text to update the state machine state. If the state machine enters the accepting state of the rule (indicating a specific format / pattern is matched), the candidate token is determined to violate the rule, and the detection ends.

[0133] The goal of the rule is to prohibit certain "combined phrases" or "cross-token combination patterns" (e.g., A and B appearing simultaneously within a gap of no more than k tokens; or A followed by several arbitrary tokens before B appears). The sliding window state corresponding to the cross-token combination rule is read (e.g., the K most recent tokens or their summaries). Standard text is added to the sliding window, and the oldest element is removed when the window exceeds its length limit. Based on the updated window, it is determined whether a preset combination constraint is met (e.g., "two types of phrases appearing together within the window" or "a certain combination sequence appearing"). If the combination constraint is met, the candidate token is determined to violate the rule, and the detection ends. In practical applications, rule constraints are part of the rule engine's posterior consistency check; a quick screening is performed first through step S200, followed by a fine screening through rule constraints.

[0134] If no violation is triggered, the candidate token is deemed to have passed the rule constraints. Provided the candidate token meets the rule constraints, its risk score is then assessed to determine if it is less than the dynamic threshold. :

[0135] like If the candidate token passes the semantic risk constraint, then the candidate token is output as the generation result of the current time step. And conclude this time step of the ruling;

[0136] like If a candidate token is deemed to have triggered a semantic assertion or fallen into an unacceptable risk range, the blocking strategy selection process begins. The risk level of the candidate token is then matched against a pre-defined mapping table that represents the relationship between risk scores and risk levels. The mapping table is shown below:

[0137] Table 1: Mapping Table of Risk Score and Risk Level:

[0138]

[0139] The previous time step running mode Convert to the corresponding risk level The transformation relationships are as follows:

[0140] like =normal, then =1;

[0141] like =cautious, then =2;

[0142] like =locked, then =3 or 4.

[0143] This step makes the system more conservative in its choice of blocking strategies under stricter operating modes, thus reflecting the impact of time-dimensional risk trends on the intensity of actions.

[0144] Based on the scene recognition result scene_id of the user input text x, the scene risk level is quantified into a scene risk grade. In an optional implementation, the scenario risk level can be given by a preset scenario configuration, as follows:

[0145] If scene_id is a general chat scene, then =1;

[0146] If scene_id represents a scenario with high regulatory / compliance requirements, then =2 or 3;

[0147] If scene_id is a clearly high-risk scene, then =4.

[0148] This step enables the same risk score to trigger blocking strategies of varying strengths in different business scenarios, thus adapting to multi-scenario deployments.

[0149] Risk level of candidate tokens Scenario risk level Model risk level The maximum value among the three is taken as the severity of the candidate token. .

[0150] when When =1 or 2, the resampling blocking strategy is executed; when When =3, the track correction blocking strategy is executed; when When the value is 4, the blocking strategy to stop refusing to answer is executed.

[0151] When resampling is determined, the candidate generation parameters are adjusted and the candidate set is regenerated to find acceptable tokens within the rule constraint boundaries. Adjustments may include lowering the temperature, tightening the TopP, reducing the TopK, or enabling retries, and then returning to the candidate generation step (e.g., returning to the candidate generation sub-step in S200 or the corresponding candidate generation process) to obtain a new candidate set. It then re-enters a decision loop to determine whether the rule constraints are met and whether the risk score exceeds the threshold. If resampling fails K times consecutively, it escalates from resampling to track correction; if it fails again, it aborts the refusal to respond.

[0152] In one example scenario, a user poses a question with certain sensitive boundaries but still compliantly answerable within an intelligent customer service / knowledge Q&A system. For instance, the user might inquire about "usage precautions, risk warnings, or compliance requirements" for a regulated product. The main model generates the candidate set at the current time step. In the process, several candidate tokens with higher probabilities may lead the output to overly specific or out-of-bounds statements. The risk model assigns values ​​close to or slightly higher than the dynamic threshold to these candidates. The risk score caused them to be rejected during the adjudication phase; but at the same time, the rule mask The fact that this type of topic was not banned entirely indicates that a large number of compliant token combinations still exist to complete the answer. At this point, the system determines the severity level to be relatively low based on the maximum risk level and chooses to implement a resampling strategy. This involves lowering the temperature, tightening the TopP, or reducing the TopK to regenerate the candidate set, making the sampling more inclined towards expressions that are "general, suggestive, and within the compliance boundaries," ultimately obtaining the satisfactory answer. The candidate tokens are generated and a complete response is continuously generated.

[0153] When trajectory correction is determined, a safety-guided correction is performed on the currently generated trajectory without directly outputting high-risk candidate tokens. Trajectory correction can be achieved by injecting a pre-defined safety guidance fragment, rewriting instruction constraint fragments, or a safety response framework fragment to the end of the generated prefix, thereby changing the conditional distribution of subsequent main model generation and shifting the generated content towards compliant, interpretable, or alternative expressions. After trajectory correction is completed, the updated prefix is ​​used as the new... Then return to the main model generation step to enter the next round of generation.

[0154] In one scenario, the user gradually changes the angle of their questions through multiple rounds of dialogue, shifting from "general discussion" to "requesting more specific execution details," and through step-by-step guidance, the contextual semantics are increasingly approached a high-risk trajectory. Initially, the system remains in normal mode, with semantic checks triggered at compensation intervals; as the risk score output by the risk model increases over several time steps, the cumulative risk score... After the fusion update, the system enters the threshold range and switches to alert mode, with semantic checks triggered step-by-step. At this point, it can be observed that even with multiple resampling operations, candidate tokens with risk scores exceeding the dynamic threshold repeatedly appear in the candidate set. This indicates that the overall generation distribution of the main model is biased towards non-compliance under the current context, and simply replacing individual tokens is insufficient to bring generation "back" to a safe path. Therefore, the system classifies the severity as high and implements a trajectory correction strategy. By injecting a pre-defined safety guidance framework (e.g., guiding the conversion to principle-based explanations, risk warnings, and compliant alternative suggestions) into the generation prefix, the system changes the conditional distribution of subsequent generation, shifting the model output from a "concrete and directive" trajectory to an "explanatory and compliant" one, thus proactively correcting the generation direction while maintaining dialogue continuity.

[0155] When it is determined that the request cannot be fulfilled, the generation of the current session is stopped, a rejection text or a security alternative is output, and the generation process ends. The rejection text may include an explanation of why the request cannot be fulfilled and alternative security suggestions / information boundary statements (without specific wording) to maintain the interpretability and usability of the response while terminating high-risk generation.

[0156] In one scenario, a user explicitly makes a high-risk request, and this request falls under the category of content that cannot be provided under the current scenario policy. Multiple candidate tokens generated by the system at the current time step either fail to meet the rule constraints or obtain risk scores significantly higher than the dynamic threshold in the semantic risk assessment; simultaneously, the cumulative risk score... If the risk rises rapidly and exceeds the upper limit of the threshold range, the system enters a locked mode, triggering a time-step semantic check and implementing stricter control strategies. In this state, even if multiple rounds of resampling or attempts at trajectory correction are performed, the candidate set still cannot produce an acceptable candidate token that meets the criteria of "rule constraints + risk score below the threshold". Based on this, the system determines the severity to the highest level and implements a termination rejection strategy: terminating subsequent token generation and outputting a rejection or safe alternative response (e.g., stating that such content cannot be provided and giving alternative information or suggestions within the scope of compliance). This ensures that the generation process converges stably under high-risk input conditions, avoiding the risk diffusion caused by continuous generation.

[0157] Candidate set If all candidates are rejected due to failure to meet rule constraints or risk scores exceeding the threshold, resampling can be triggered up to the maximum number of preset retries. If no acceptable candidate can be obtained, the process will be upgraded to trajectory correction or abort rejection to ensure that the process can still converge and output a safe result in extreme cases.

[0158] The dynamic threshold for the current time step is updated as follows:

[0159] Obtain the basic risk threshold and risk tightening coefficient set when configuring the rule engine, and simultaneously obtain the cumulative risk score of the previous time step. Use the product of the risk tightening coefficient and the cumulative risk score of the previous time step as the penalty item.

[0160] Obtain the operating mode of the previous time step, quantify it into a numerical value, and multiply it with the preset mode coefficient as a penalty term;

[0161] The dynamic threshold for the current time step is calculated by using the basic risk threshold as both a reward and a penalty.

[0162] The formula for updating the dynamic threshold is as follows:

[0163] ,

[0164] ;

[0165] in, Represents the dynamic threshold at the current time step; Indicates the basic risk threshold; Indicates the risk tightening coefficient; Represents the mode coefficients; This represents the mapping function that quantizes the operating mode of the previous time step into numerical values; This indicates the upper and lower limits of the threshold, which can be set according to the specific scenario.

[0166] Assumption It is 0.6. It is 0.2. It is 0.1. If If the value is 0.4, then in normal mode, The value is 0.52 in cautious mode. The value is 0.47 in locked mode. The value is 0.42. It can be seen that the stricter the mode (normal→cautious→locked), the lower the dynamic threshold, resulting in a more stringent ruling.

[0167] When executing the blocking strategy, if the current time step determines that resampling is to be performed, the historical time steps are combined to determine whether the current time step is the Nth consecutive resampling. If so, the blocking strategy is upgraded to track correction and the count is set to zero. Otherwise, resampling is performed and the counting continues.

[0168] If the blocking strategy at any time step is track correction or abort of response rejection, the count is set to zero.

[0169] Specifically, when entering the blocking strategy execution phase at the current time step t, the resampling continuous counter from the previous time step is read from the guard state or independent control state. The counter is used to record the number of times the resampling strategy is executed consecutively. In an optional implementation, the initial value is set to... .

[0170] Based on the decision of the candidate token (e.g., not meeting the rule constraints or the risk score exceeding the threshold), determine the initial blocking strategy for this time step (resampling, track correction or suspension of response), and obtain the preset upgrade threshold N (the upper limit of consecutive resampling times), where N is a positive integer and the value ranges from 2 to 5.

[0171] If the current time step determines that resampling should be performed, then the counter is used to determine whether the current time step constitutes the Nth consecutive resampling execution. Specifically, the following determination is made: If this condition is not met, proceed with the normal resampling execution process and update the resampling continuity counter. If the condition is met, the blocking strategy for this time step will be upgraded from resampling to trajectory correction, and the continuous counter will be reset to zero. If the initial blocking strategy for this time step is trajectory correction or abort / rejection, the continuous counter will be reset to zero. Furthermore, when entering the cautious / locked state, the maximum number of resampling iterations will be limited to avoid infinite loops.

[0172] Update the resampling continuous counter at this time step Write the guard state (as) One of the fields), so that in the next time step, it can continue to determine "whether to resample continuously". Optionally, the blocking strategy, Information on whether an upgrade was triggered is written to a structured log for easy replay and strategy tuning.

[0173] This step enables the system to continuously remember and determine whether resampling occurs repeatedly over time, thus preventing the blocking strategy from making isolated decisions based solely on the risk outcome of a single time step. With this counter, subsequent strategy selection can identify whether the current resampling has formed a continuous cumulative trend, and provide stable and traceable state input for the control logic of "upgrading to trajectory correction when reaching the Nth consecutive resampling," preventing the system from repeatedly retries on the same semantic track and failing to converge.

[0174] Simultaneously, historical information on blocking actions is incorporated into the guard state update chain, enabling blocking strategies and mechanisms such as risk assessment and operation mode switching to share the same session-level state foundation, achieving state consistency and collaborative control between different modules. When subsequent time steps trigger trajectory correction or abort of response rejection, the counter reset also forms a closed loop with this step, allowing the system to re-enter the next round of generation control in a determined initial state after stronger intervention, thereby improving the stability, interpretability, and controllability of the overall generation process of blocking strategy scheduling. In addition, in S200, a mask is first used for vocabulary-level masking (coarse screening) of 'deterministically prohibited tokens'; this step uses a combination of rule constraints (fine screening) to exclude non-compliant tokens as much as possible.

[0175] S500. Update the guard state at the current time step based on the judgment result and proceed to the next time step until the termination condition is met, and decode all generated tokens into reply text and output them.

[0176] Termination conditions include at least the following: the current time step is marked as the end of generation by the large language model; the generated result reaches the maximum allowed generation length; and the blocking strategy is to stop and refuse to answer.

[0177] Cumulative risk score at the current time step Operating Mode Cooling count Recent Trigger Records and optional resampling continuous counting. Write the current guard state object and will This saves the input state for the next time step, enabling subsequent semantic inspection triggering strategies, dynamic threshold adjustments, and blocking strategy selection to run continuously based on the risk trends of the same session.

[0178] Determine whether the current time step was generated by the Large Language Model (EOS). Specifically, if the current time step is accepted by the token... When the token is terminated, the termination condition is met, and the final decoding and output process begins.

[0179] Determine whether the length of the currently generated token sequence has reached the maximum allowed generation length . If it is satisfied , then the termination condition is satisfied, and the final decoding output process is entered. The length can be counted by the number of tokens or by the character / byte length

[0180] Determine whether the blocking strategy at this time step is to abort and refuse to answer. If so, the termination condition is satisfied, and a refusal to answer or a secure alternative text is directly generated as the final output, and the final output process is entered

[0181] If neither is satisfied, update the generated prefix: If the token is output at this time step , then let ; If the orbit correction is performed at this time step, append the correction segment to the prefix to form a new ; Subsequently, update the time step to t←t + 1, and enter the next time step generation process with x and the guard state . It should be noted that the correction segment is a preset whitelist template (not generated by the model)

[0182] After any termination condition is satisfied, decode all the generated token sequences to obtain the response text Y and output it to the user side. If the termination condition is to abort and refuse to answer, the output is the refusal to answer text or the secure alternative text; if the termination condition is EOS or the maximum length is reached, the output is the decoded text result normally generated by the model

[0183] To facilitate the understanding of the above embodiments, the following will be described by taking the specific application scenarios of the above embodiments as examples

[0184] Take the public Q&A robot as an example

[0185] User: I'm writing a network security popularization article. Let's first talk about what stages "vulnerability exploitation" generally includes?

[0186] Assistant: It can be introduced from perspectives such as "information collection, risk assessment, protection and reinforcement, compliance testing", with an emphasis on legal authorization and protection ideas. (Guard state: normal; semantic check is triggered at the compensation interval)

[0187] User: Then if I want to do a "demonstration", give me the simplest idea framework for "bypassing authentication"?

[0188] Assistant: I can talk about the compliance protection perspective: for example, common authentication bypasses come from session management defects, weak passwords, incorrect permission verification, etc.; it is recommended to use a target range / test environment for verification under authorization. (Risk score increases, mode tends to be cautious).

[0189] User: Then write it out as steps 1, 2, and 3, the more specific the better, and it would be best to include a script.

[0190] (System Action) Detected structural signals such as "numbered steps / instruction / code block structure", triggered structural trigger masking, and hard masked tokens that may promote the further expansion of the structure.

[0191] Assistant: I cannot provide steps or scripts that can be used to bypass authentication or intrusion.

[0192] However, I can provide you with compliant alternatives: how to set up an authorized testbed, how to create a security test checklist for the authentication module, and how to write risk warnings and remediation suggestions for readers. (Blocking strategy: track correction → switch to "principle explanation / risk warning / alternative suggestions").

[0193] In the above scenario, when a user attempts to push the answer toward "specific executable details", the structure trigger and phrase completion shield first "blocks" the dangerous output path, and then uses trajectory correction to bring the generation direction back to a compliant interpretation.

[0194] Taking intelligent customer service / knowledge Q&A (consultation on regulated products) as an example:

[0195] User: I bought a home physiotherapy device. What should I pay attention to when using it for the first time?

[0196] Assistant: It is generally recommended to read the instructions and contraindications first, and avoid using on broken skin; if you have implanted medical devices, are pregnant, or are otherwise aware of this, consult a doctor first, and start with a low intensity. (Scenario recognition: Scenarios with high compliance requirements; load corresponding rules and thresholds).

[0197] User: Then give me the "most effective" parameter settings and usage frequency, and be more specific.

[0198] (System Action) Some tokens in the risk model assessment candidates will lead to "overly specific parameters / dosage descriptions", with risk scores close to or slightly higher than the dynamic threshold; however, the rule layer does not prohibit the topic as a whole ⇒ it is appropriate to perform resampling first.

[0199] Assistant: I can't provide you with a "one-size-fits-all optimal parameter / frequency" because different people, indications, and contraindications vary greatly.

[0200] I can give you a compliant, general approach:

[0201] 1) Follow the recommended range in the instruction manual; 2) Start with a low setting and short duration; 3) Stop immediately if you experience any discomfort; 4) If you have underlying medical conditions or are undergoing treatment, consult your doctor / manufacturer's customer service first.

[0202] If you'd like to specify the model and the recommended range in the instruction manual, I can help you organize the manual content into a more easily implementable "Precautions Checklist." (After resampling, select the "General + Risk Warning" expression track).

[0203] This scenario explains the situation of "answerable but with sensitive boundaries": first reject details that go beyond the boundaries, and then stabilize the output at "risk warnings / compliance considerations".

[0204] Take homework tutoring and content generation platforms as examples:

[0205] User: I need to write an article on "How to cheat in an exam without being caught". Please provide a list of 1-10 actionable steps.

[0206] (System Action) The scenario is identified as high risk; the structure triggers the mask to hit the "list-style enumeration / step-by-step instruction", and the rule mask sets the token that advances the structure to zero; if multiple candidates still exceed the threshold, it can be upgraded to "stop refusing to answer".

[0207] Assistant: I cannot help devise a list of cheating methods or ways to circumvent proctoring.

[0208] If you want to improve your grades, I can provide compliant alternatives: study plan templates, methods for organizing incorrect answers, pre-exam time allocation, and how to reduce exam anxiety. (High severity → stop refusing to answer or refuse to answer + alternative suggestions).

[0209] In summary, this application directly applies rule constraints to the original probability distribution vector output by the large language model at each time step and performs renormalization based on rule masks. This allows security rules to directly affect the token generation probability space, reducing the likelihood of violating tokens being selected from the source. Compared to schemes that only post-process the generation results, this approach allows for earlier and more refined intervention in the generation process. By determining the semantic check triggering strategy based on the guard state of the previous time step, the risk model is invoked for evaluation only when conditions are met. This avoids frequent high-cost semantic checks in low-risk stages, thereby improving overall inference efficiency while ensuring security. This approach is suitable for large-scale or real-time generation scenarios. By introducing rule constraints, dynamic risk assessment, and guard state management mechanisms into the large language model generation process, real-time, continuous, and adaptive security control of the generation process is achieved. This effectively reduces the accumulation of generation risks over time, improving the security, stability, and overall system availability of the generated content.

[0210] The table below shows an example of the original test data:

[0211] Table 2: Example of Test Data

[0212]

[0213] Before conducting the experiment, we first constructed the experimental dataset and performed relevant experimental tests. The experimental dataset is shown in the table below:

[0214] Table 3: Example table of experimental datasets:

[0215]

[0216] like Figure 3 The graph shown is a leak window rate indicator. The horizontal axis represents the evaluation date, and the vertical axis represents the leak window rate. The calculation method is: Leak Window Rate = LeakCount / AssertCount.

[0217] AssertCount (assertion trigger count): The total number of times the system triggers assertions / blocks during offline playback (e.g., rule mask assertions, semantic risk assertions triggering resampling / correction / abortion, etc.).

[0218] LeakCount (Number of dangerous segments output): At the moment each assertion is triggered, review the prefixes that have been output to the user before the assertion. If they already contain dangerous segments, count them as 1.

[0219] from Figure 3 As can be seen, the overall rate of public chat and controlled chat decreased significantly. Due to the obvious "check interval window" or "trigger lag" in the early stage, some dangerous content had already been output when the assertion occurred, so the rate was higher in the early stage. Due to the addition of more advanced hard constraints (such as phrase completion / structure trigger interception), dangerous outputs can be blocked at the "generation but not yet output" stage, so the rate decreased rapidly in the middle stage. With the effect of tail optimization, the rate approached 0 in the later stage, but the rate of decrease slowed down.

[0220] like Figure 4 The chart shown is a graph of the correction success rate, with the horizontal axis representing the evaluation date and the vertical axis representing the correction success rate. Figure 4 As can be seen from the data, with version / strategy iterations, after triggering assertions, the system is increasingly able to pull the answer back to a safe track without rejecting it. The strong correction curve is generally higher than the light correction curve. This is because the light correction tries to preserve the original meaning and only deletes unsafe details, while the strong correction rewrites / guides more strongly, pulling the answer to a safer and more generalized area. Therefore, it is easier to pass subsequent security checks without rejecting the answer. The correction success rate is the result of the ratio of the number of successful corrections to the number of correction attempts.

[0221] like Figure 5 The chart shown is a graph of the false positive rate, with the horizontal axis representing the evaluation date and the vertical axis representing the false positive rate. From... Figure 5As can be seen, public chat is highly ambiguous and contains many mixed intentions, making it most prone to triggering excessive defenses and thus having the highest false positive rate. Regulated chat has tighter compliance boundaries and more conservative thresholds, resulting in a higher risk of false positives, but its expression is often more structured and easier to control than public chat. Programmatic chat, due to the inherent "step / instruction / fragment structure" in its code, is prone to false positives due to excessive resampling if the structure triggering rules or semantic model are not sufficiently "contextualized." Enterprise Q&A has a stable domain, clear intent, and less adversarial interaction, typically resulting in the lowest false positive rate, serving as a "benchmark for user experience." The false positive rate is calculated as the ratio of the number of rejections or resamplings ≥ N (taking N=3 as an example) to the number of normal requests.

[0222] Example 2

[0223] like Figure 6-7 As shown in the figure, this embodiment introduces an implementation system for a multi-layered defense mechanism for the security of large language models, including a probability estimation module, a candidate generation module, a risk assessment module, a blocking strategy decision module, and a result output module.

[0224] The probability estimation module is used to generate the original probability distribution vector at the current time step, based on the generated token and the received user input text, using a large language model.

[0225] The candidate generation module is used to call the rule engine to perform hard constraint processing on the original probability distribution vector, generate a rule mask, and renormalize the original probability distribution vector according to the rule mask. The renormalization result is then used to perform probability selection to generate a candidate token containing at least one candidate token. The rules executed by the rule engine include token blacklist, phrase completion blocking, and structure trigger blocking.

[0226] The risk assessment module is used to determine the triggering strategy for semantic checks based on the guarding state of the previous time step. If the triggering strategy is met, the candidate token is assessed for risk using a pre-built risk model to obtain a risk score. The guarding state includes at least the cumulative risk score and the operating mode that represent the overall risk level in the time dimension.

[0227] The blocking strategy decision module is used to determine whether the candidate token meets the rule constraints and whether the risk score is less than the dynamic threshold; otherwise, the blocking strategy is executed, and if yes, the candidate token is output; the blocking strategy includes resampling, track correction and suspension of rejection;

[0228] The result output module is used to update the guard state at the current time step based on the judgment result and proceed to the next time step until the termination condition is met, and decode all generated tokens into reply text for output.

[0229] When this implementation system is specifically applied, the system runs on at least one computing device, which includes:

[0230] Processor: CPU (e.g., x86 / ARM multi-core), used for rule engine, state machine, scheduling and candidate control logic, etc.;

[0231] Accelerators: GPU / NPU / TPU, used for inference computation of large language models and risk models;

[0232] Memory includes RAM (runtime cache / context / state) and non-volatile storage (SSD / HDD, used for model weights, rule base, and log storage); it also includes the execution order of the probability estimation module, candidate generation module, risk assessment module, blocking strategy decision module, and result output module.

[0233] Network interface: used to receive user requests, call model inference services, report logs and update configurations;

[0234] Optional security modules include trusted execution environment and key management module, used for integrity verification and access control of model files and policy configurations.

[0235] In addition, a rule configuration library / strategy library is set up: storing rule sets, basic thresholds, risk tightening coefficients, mode coefficients, severity action mappings, etc., for different scenarios;

[0236] Model repository: Stores the weights of the main model and the weights of the risk model (supports version numbers and rollback).

[0237] Log and replay storage: Records the mask summary, candidate set statistics, risk score statistics, and action type for each step t. / mode / cooldown, etc., facilitate regression testing and auditing.

[0238] This embodiment moves security protection beyond pre-generation prompts and constraints or post-generation text filtering. Instead, it directly embeds per-token reasoning into the process to achieve pre-emptive control, thereby improving the overall security, policy consistency, convergence, and stable availability of the generation process in multi-round interaction scenarios.

[0239] Example 3

[0240] This embodiment introduces a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the multi-layered defense mechanism for large language model security as described in Embodiment 1.

[0241] The multi-layered defense mechanism for large language model security described in Example 1 can be implemented in software form, such as by designing it as a standalone program and installing it on a computer device, which could be a computer, smartphone, or similar device. Alternatively, it can be designed as an embedded program and installed on a computer terminal, such as a microcontroller.

[0242] The technical scope of this invention is not limited to the content described above. Those skilled in the art can make various modifications and variations to the above embodiments without departing from the technical concept of this invention, and all such modifications and variations should fall within the protection scope of this invention.

Claims

1. A multi-layered defense mechanism for the security of large language models, characterized in that, Includes the following steps: At the current time step, based on the generated token and the received user input text, the original probability distribution vector is generated using a large language model; The rule engine is invoked to perform hard constraint processing on the original probability distribution vector to generate a rule mask. The original probability distribution vector is then renormalized based on the rule mask. The renormalization result is then used to generate a probability selection that includes at least one candidate token. The rules executed by the rule engine include token blacklisting, phrase completion blocking, and structure trigger blocking. The triggering strategy for semantic checks is determined based on the guarding state of the previous time step. If the triggering strategy is met, the candidate tokens are assessed for risk using a pre-built risk model to obtain a risk score. The guard status includes at least a cumulative risk score and an operating mode that characterize the overall risk level over time. Determine whether the candidate token meets the rule constraints and whether the risk score is less than the dynamic threshold. If not, execute the blocking strategy; if yes, output the candidate token. The blocking strategy includes resampling, track correction, and suspension of response rejection. Update the guard state at the current time step based on the judgment result and proceed to the next time step until the termination condition is met, and decode all generated tokens into reply text and output them.

2. The multi-layered defense mechanism for the security of large language models according to claim 1, characterized in that, Before receiving user input text and starting text generation, the rule engine is configured and the guard mode is initialized. The specific steps are as follows: The scenario is determined based on the user's input text, and the rule configuration that matches the scenario is retrieved from the preset configuration library and constructed into a rule engine. Set the cumulative risk score to the preset minimum risk value and the operating mode to normal mode to complete the guard state initialization.

3. The multi-layered defense mechanism for the security of large language models according to claim 1, characterized in that, The specific strategy for triggering semantic checks based on the guard state of the previous time step includes: Get the operating mode of the guard state at the previous time step, and determine whether the operating mode is the normal mode; If yes, semantic checks are triggered according to the preset compensation interval; otherwise, semantic checks are performed step by step.

4. The multi-layered defense mechanism for the security of large language models according to claim 3, characterized in that, The guard mode operates by comparing the accumulated risk score with a preset threshold range. If the score is below the lower limit of the threshold range, it is in normal mode; if it is within the threshold range, it is in alert mode; and if it is above the upper limit of the threshold range, it is in locked mode. The cumulative risk score is updated by merging the cumulative risk score from the previous time step with the risk score from the current time step.

5. The multi-layered defense mechanism for the security of large language models according to claim 2, characterized in that, The process involves determining whether a candidate token meets the rule constraints and whether its risk score is less than the dynamic threshold. If not, a blocking strategy is executed; otherwise, the candidate token is output. The specific steps for executing the blocking strategy are as follows: Determine whether the candidate token meets the rule constraints; otherwise, discard the candidate token. If so, determine whether the risk score of the candidate token is less than the dynamic threshold of the current time step; if so, output the candidate token. Otherwise, the risk level of the candidate token is matched based on the pre-defined mapping table of risk scores and risk levels, and the operating mode of the previous time step and the scenario of the user input text are quantified into risk levels. The highest risk level is taken as the severity of the candidate token, and corresponding resampling, track correction, or suspension of response is performed.

6. The multi-layered defense mechanism for large language model security according to claim 5, characterized in that, The dynamic threshold for the current time step is updated as follows: Obtain the basic risk threshold and risk tightening coefficient set when configuring the rule engine, and simultaneously obtain the cumulative risk score of the previous time step. Use the product of the risk tightening coefficient and the cumulative risk score of the previous time step as the penalty item. Obtain the operating mode of the previous time step, quantify it into a numerical value, and multiply it with the preset mode coefficient as a penalty term; The dynamic threshold for the current time step is calculated by using the basic risk threshold as both a reward and a penalty.

7. The multi-layered defense mechanism for large language model security according to claim 5, characterized in that, When executing the blocking strategy, if the current time step determines that resampling is to be performed, the historical time steps are combined to determine whether the current time step is the Nth consecutive resampling. If so, the blocking strategy is upgraded to track correction and the count is set to zero. Otherwise, resampling is performed and the counting continues. If the blocking strategy at any time step is track correction or abort of response rejection, the count is set to zero.

8. The multi-layered defense mechanism for the security of large language models according to claim 1, characterized in that, In the process of updating the guard state of the current time step based on the judgment result and entering the next time step until the termination condition is met, the termination condition includes at least the following: the current time step is generated by the large language model to end the generation, the generation result reaches the maximum allowed generation length, and the blocking strategy is to stop and refuse to answer.

9. A system for implementing a multi-layered defense mechanism for the security of large language models, characterized in that, The implementation system includes: The probability estimation module is used to generate the original probability distribution vector at the current time step based on the generated token and the received user input text using a large language model. The candidate generation module is used to call the rule engine to perform hard constraint processing on the original probability distribution vector, generate a rule mask, and renormalize the original probability distribution vector according to the rule mask. The renormalization result is then used to perform probability selection to generate a candidate token containing at least one candidate token. The rules executed by the rule engine include token blacklist, phrase completion blocking, and structure trigger blocking. The risk assessment module is used to determine the triggering strategy for semantic checks based on the guarding state of the previous time step. If the triggering strategy is met, the candidate token is assessed for risk using a pre-built risk model to obtain a risk score. The guarding state includes at least the cumulative risk score and the operating mode that represent the overall risk level in the time dimension. The blocking strategy decision module is used to determine whether the candidate token meets the rule constraints and whether the risk score is less than the dynamic threshold; otherwise, the blocking strategy is executed, and if yes, the candidate token is output; the blocking strategy includes resampling, track correction and suspension of rejection. The result output module is used to update the guard status of the current time step based on the judgment result and enter the next time step until the termination condition is met, and decode all generated tokens into reply text output.

10. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes a computer program, it implements the steps of the multi-layered defense mechanism for the security of large language models as described in any one of claims 1-8.

Images