A large model security detection method and system based on cross-layer reliability fusion

Abstract

The application discloses a large model security detection method and system based on cross-layer reliability fusion. The text to be detected is detected by rules, vectors and semantics in turn, and any layer risk score exceeding the dangerous threshold is intercepted. When none of the layers is intercepted, the heterogeneous risk scores of each layer are calibrated and mapped into calibrated risk probabilities, the input reliability is derived from the uncertainty of the current input by each layer, and the basic weight is modulated to obtain the cross-layer cumulative risk score. When the cumulative score is not less than the cumulative threshold, or the divergence between layers is not less than the divergence threshold, and the maximum calibrated probability is not less than the lower limit, the request is upgraded to the asynchronous final review based on the large model. The application can compare and adaptively fuse the multiple detection signals with different dimensions and variable reliabilities, and can consider low delay, high coverage and low false alarm, and prevent privacy leakage through input and output bidirectional desensitization.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence security technology, specifically to a large-scale model security detection method and system based on cross-layer reliability fusion. Background Technology

[0002] With the rapid development of Large Language Model (LLM) technology, its application in scenarios such as intelligent customer service, code generation, content creation, and knowledge-based question answering is becoming increasingly widespread. While bringing powerful capabilities, large models also face severe security challenges. Currently, large models mainly face three core security threats: prompt word injection attacks, adversarial example attacks, and privacy data leakage. Prompt word injection attacks bypass the model's security restrictions and instruction alignment mechanisms through carefully crafted input text, inducing the model to output content that should be prohibited. For example, they hijack model behavior by embedding system-level instructions in user input or using special character separators. Adversarial example attacks circumvent traditional keyword matching detection by using synonym replacement, variant expressions, mixed pinyin, character shape transformation, and encoding conversion to manipulate sensitive keywords, rendering keyword detection systems based on exact matching ineffective. Privacy data breaches refer to situations where user-input text may contain personally identifiable information (PII), such as mobile phone numbers, ID card numbers, bank card numbers, and email addresses. If this information is not anonymized and directly transmitted to the large model server, it will lead to serious privacy breach risks. When the large model is called via a public application programming interface (API), sensitive data is transmitted over public networks, further amplifying the possibility of leakage.

[0003] Existing protection solutions against the aforementioned threats are mostly single-point detection methods, resulting in a significant trade-off between performance and coverage. Solutions relying solely on rule engines employ AC automata multi-pattern string matching algorithms, regular expressions, and Bloom filters for keyword matching and rapid pre-filtering, achieving an overall coverage of only about 40%. They cannot identify adversarial attacks such as variant words and synonym substitutions, nor can they detect semantically sensitive content. Solutions relying solely on vector semantic matching map text to high-dimensional vectors and utilize vector similarity to detect variant words and semantically similar sensitive content, achieving a coverage of about 65%. However, their generalization ability to new attack patterns is limited, and vector retrieval suffers from high latency and a certain false positive rate. Solutions relying solely on small-model detection utilize lightweight deep learning models for semantic risk classification, achieving a coverage of about 85%. However, their classification ability is limited by training data, and they still miss attack patterns outside the training distribution. Solutions relying solely on large-model detection utilize large-scale security content review models for review, achieving 100% coverage. However, their latency is over 200 milliseconds, costs are high, and there is a serious risk of privacy leakage when accessed via the public internet.

[0004] Existing multi-layered defense schemes typically employ a cascaded structure where any layer determines danger and intercepts, with each layer making independent decisions, or a fusion decision method that compares a weighted average of the risk scores from each layer with a single threshold. These schemes have the following drawbacks: First, the output dimensions of each detection layer are inconsistent. The severity heuristic score from the rule engine, the cosine similarity from vector retrieval, and the normalized confidence score from the detection model have different dimensions and distributions, making direct weighted summation incomparable. Second, the fixed weights for each layer fail to reflect the reliability of a particular detection layer for a specific input. Fixed weights still adopt scores according to a predetermined proportion even when the layer is highly uncertain about the current input, thus introducing noise. Third, the weighted average calculation discards information about the dispersion of decisions made by each layer. When a new type of evasion attack... When an attack generates a strong signal only in a single detection layer while other layers are deemed safe, the cumulative score after weighted averaging may be lowered below the threshold, causing it to be missed. Lowering the threshold across the board to capture such attacks would increase false positives overall. Fourth, training smaller models using larger models indiscriminately affects samples, resulting in low sample efficiency. Furthermore, the weights and thresholds used for fusion gradually become inaccurate due to attack distribution shifts without correction. Fifth, sensitive information desensitization only applies to the input side, while the larger model may still echo placeholder strings or reconstruct the original personal identity information based on multiple rounds of context when generating responses, thus bypassing input-side desensitization. Therefore, there is an urgent need for a large-model security protection scheme that can balance low latency, high coverage, strong privacy protection, and rigorous and adaptive fusion of heterogeneous and variable-reliability-per-input multi-detection signals. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention proposes a large-model security detection method and system based on cross-layer reliability fusion, comprising:

[0006] The system receives large model interaction requests and extracts the text to be detected. The text is then sequentially input into multiple cascaded detection layers. Each detection layer includes at least a rule-based detection layer, a vector-based detection layer, and a semantic detection layer based on a lightweight detection model. Each detection layer outputs a normalized risk score. And in If the request exceeds the danger threshold of the detection layer, it is blocked. If none of the detection layers block the request, the following method is used to determine whether to escalate the request to the asynchronous final review layer based on the security content review model: Each The calibration risk probability is transformed into a uniform dimension through a preset calibration mapping. ; The input-wise reliability of each detection layer is derived from the uncertainty signal of the current text to be detected. , preset basic weights and Multiply and normalize to obtain the input-wise effective weights ,according to Calculate the cross-layer cumulative risk score and categorize it according to the calibrated risk probability. The inter-layer divergence degree D is calculated relative to the dispersion of the cross-layer cumulative risk score; when Greater than or equal to the preset cumulative threshold Or when the interlayer divergence degree D is greater than or equal to a preset divergence threshold. And each The maximum value in the range is not less than the preset lower limit. If the request is deemed suspicious, it is deemed safe otherwise.

[0007] Furthermore, the calibration mapping is obtained by fitting historical detection logs with the final review result or manual review label of the asynchronous final review layer as the truth value. Temperature calibration is used for the semantic detection layer, and ordinal-preserving regression or Platt calibration is used for the rule detection layer and the vector detection layer. The historical detection logs are divided into training set and validation set according to time sequence to avoid time leakage. The inter-layer divergence degree D represents the dispersion of the calibration risk probability of each detection layer and is calculated according to the weighted variance of each calibration risk probability relative to the cross-layer cumulative risk score.

[0008] Furthermore, the determination of the per-input reliability includes: the reliability of the rule detection layer is determined by the coverage of the text to be detected relative to the risk word library; the higher the proportion of out-of-vocabulary words in the text to be detected, or the lower the reliability of the rule detection layer when the rule detection layer does not produce any hits or near-hits; the reliability of the vector detection layer is determined by the difference between the highest and second-highest cosine similarity in its Top-K retrieval results, or the distance between the highest cosine similarity and the judgment threshold; the larger the difference between the highest and second-highest cosine similarity, or the distance between the highest cosine similarity and the judgment threshold, the higher the reliability; the reliability of the semantic detection layer is determined by the variance or prediction entropy of the risk scores output by the multiple detection units concurrently called by it; the larger the variance or prediction entropy, the lower the reliability.

[0009] Furthermore, the cumulative threshold is adaptively adjusted according to the interlayer divergence degree, i.e., based on... The alternative to the cumulative risk criterion ,in A non-negative coefficient is preset so that the greater the degree of divergence between the layers, the lower the threshold for upgrading the asynchronous final review layer.

[0010] Furthermore, the basic weights of each detection layer The distribution of weights increases asymmetrically from the rule detection layer to the semantic detection layer, meaning the base weights of the rule detection layer are less than the base weights of the vector detection layer, and the base weights of the vector detection layer are less than the base weights of the semantic detection layer; the base weights... The cumulative threshold The divergence threshold and the lower limit Historical detection logs based on the deployment environment are obtained through post-event statistical analysis and are configured independently for different business scenarios.

[0011] Furthermore, the method also includes selective knowledge distillation oriented towards the decision boundary, performed from the large-scale security content moderation model to the lightweight detection model of the semantic detection layer: from the samples finalized by the asynchronous final review layer, the distillation weights of each sample are determined according to the difference between the risk probability output by the large-scale security content moderation model and the calibration risk probability output by the semantic detection layer, or according to the per-input reliability of the semantic detection layer for that sample, so that samples with a larger difference in calibration risk probabilities or lower per-input reliability receive higher distillation weights, and the lightweight detection model is incrementally trained after weighting the distillation loss with the distillation weights; and periodically, the calibration mapping is refitted and the basic weights are re-estimated using the final review result of the asynchronous final review layer as the truth value. The decision-making mechanism on which the update and upgrade judgment are based will be updated.

[0012] Furthermore, the incremental training employs a catastrophic forgetting constraint mechanism, which includes at least one of the following: empirical replay of sampling from the historical sample pool and mixing it with the current day's samples using attack category equalization and time decay weighting; introducing a flexible weight solidification regularization term based on the Fisher information matrix of the previous stable version parameters in addition to the distillation loss; maintaining a historical guardian evaluation set isolated from the training data, and rejecting deployment and rolling back to the previous stable version when the recall rate of the new model on any historical attack category in the historical guardian evaluation set decreases by more than a preset threshold relative to the previous version; performing distribution drift detection on the current day's samples and the historical sample pool, and pausing the incremental training when the drift index exceeds a preset threshold.

[0013] Furthermore, before sending the text to be detected to the security content review big model and before allowing the request to the target big model service, the text to be detected is subjected to format-preserving sensitive information replacement: sensitive information entities in the text to be detected are identified and replaced. The length of the replaced placeholder string is not greater than the length of the original sensitive entity and does not use symbols that will be parsed as structured tags by the big model. The original value before replacement and the desensitized value after replacement are stored in the cache database with the session identifier as the group key to form a session-level mapping relationship. Multiple replacements of the same original value within the same session return idempotent placeholders. The session-level mapping relationship is set with an expiration time aligned with the session lifecycle.

[0014] Furthermore, the method also includes intercepting sensitive information leakage on the output side: after the target large model service returns a response but before returning the response to the client, reverse detection is performed on the response: for the placeholder string in the response corresponding to any de-identified placeholder in the session-level mapping relationship, only authorized callers are restored to the original value according to the session-level mapping relationship; for sensitive information in the response that does not have a corresponding original value in the session-level mapping relationship, it is determined to be unauthorized leakage and intercepted or re-de-identified, and an audit log is recorded; and for any text sent to the security content review large model and the target large model service, it is verified according to the session-level mapping relationship that it does not contain any original sensitive information value before replacement.

[0015] Furthermore, the risk score of the vector detection layer is also determined by combining the projection energy of the risk subspace: a set of risk direction vectors or risk prototype vectors obtained through comparative learning are preset for each risk category to span the risk subspace corresponding to the risk category, and the projection energy of the vector of the text to be detected on each risk subspace is fused with the similarity retrieval result of the vector detection layer, and the fusion result is taken as the risk score of the vector detection layer.

[0016] A method and system for large-scale content security detection based on cross-layer reliability fusion, used to implement the aforementioned method for large-scale content security detection based on cross-layer reliability fusion, includes: a request access module for receiving large-scale model interaction requests and extracting the text to be detected; multiple cascaded detection modules, including at least a rule detection module based on keyword and pattern matching, a vector detection module based on vector semantic similarity retrieval, and a semantic detection module based on a lightweight detection model, each detection module outputting a normalized risk score and outputting a dangerous state to intercept the request when the risk score exceeds a danger threshold; and an asynchronous final review module for performing deep semantic analysis and intent recognition on upgrade requests based on the large-scale security content review model; characterized in that the system further includes a cross-layer cumulative decision module, which is connected to the rule detection module, vector detection module, semantic detection module, and asynchronous final review module respectively, and configured as follows: The risk scores output by each detection module are transformed into a calibration risk probability with a unified probability dimension through a calibration mapping that takes the final review result of the asynchronous final review module or the manual review label as the true value and is fitted based on historical logs. The per-input reliability of each detection module is derived from the uncertainty signal of the judgment of the current text to be detected by each detection module. The per-input reliability is used to modulate and normalize the preset basic weights of each detection module to obtain the per-input effective weights, and the cross-layer cumulative risk score is calculated accordingly. When the cross-layer cumulative risk score is greater than or equal to a preset cumulative threshold, or when the inter-layer divergence is greater than or equal to a preset divergence threshold and the maximum value of each calibration risk probability is not less than a preset lower limit, the request is routed to the asynchronous final review module; otherwise, the request is deemed safe. The inter-layer divergence is calculated based on the weighted variance of each calibration risk probability relative to the cross-layer cumulative risk score.

[0017] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention provides a large model security detection method and system based on cross-layer reliability fusion. It makes heterogeneous scores comparable by cross-layer probability calibration, and modulates the weights of the input reliability to amplify the judgment of the reliable layer and suppress the noise of the unreliable layer, thereby improving the accuracy without increasing the proportion of the fourth layer.

[0018] 2. This invention provides a large-scale model security detection method and system based on cross-layer reliability fusion. By upgrading the dual criteria of cumulative criteria and inter-layer divergence criteria, it can specifically restore coverage of single-layer strong signal-type covert attacks without lowering the overall threshold.

[0019] 3. This invention provides a large model security detection method and system based on cross-layer reliability fusion. By selective distillation oriented towards the decision boundary and periodic recalibration of the fusion mechanism, the feedback from the large model to the small model is more efficient in terms of sample size, and the fusion mechanism self-corrects as the distribution drifts.

[0020] 4. This invention provides a large-scale model security detection method and system based on cross-layer reliability fusion, which fundamentally solves the privacy leakage problem in public network call scenarios by maintaining desensitization through bidirectional formatting on the input and output sides. Attached Figure Description

[0021] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of the overall architecture of the present invention; Figure 2 This is a flowchart of the four-layer safety protection process of the present invention; Figure 3 This is a flowchart of cross-layer reliability calibration, dynamic weighting, and dual-criteria upgrade process; Figure 4 This is a schematic diagram of the selective distillation and fusion self-calibration closed loop; Figure 5 This is a schematic diagram of the two-way sensitive information desensitization fence data flow. Detailed Implementation

[0023] The technical solution of the present invention will be more clearly and completely explained below with reference to the accompanying drawings and through the description of preferred embodiments of the present invention.

[0024] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings. The following embodiments are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0025] The system of this invention is deployed between the API gateway and the backend large model service, running as an independent security protection middleware. Its overall architecture is as follows: Figure 1As shown, the processing flow is as follows: A large model interaction request is received from the client; API authentication and business scenario matching are performed on the request; after successful authentication and scenario matching, the text to be detected and context information in the request are extracted; the text to be detected is sequentially input into a rule detection layer, a vector detection layer, and a semantic detection layer for simultaneous detection in three layers. Each detection layer outputs a risk score normalized to the range of 0 to 1. If the risk score of any detection layer exceeds the danger threshold of that layer, it determines danger, intercepts the request, and returns a security blocking response. If no interception is detected in the three-layer synchronous detection, the risk score of the three layers is reliably calibrated by cross-layer cumulative decision-making, dynamically weighted for each input and comprehensively judged by dual criteria to determine whether to upgrade the request to the fourth layer asynchronous final review based on the security content review big model; Finally, before the results are released to the target large model service, format-preserving sensitive information replacement is performed, and output-side leakage interception is performed on the response returned by the target large model service. Based on the final judgment result, release, interception, or post-event processing is performed, and the response is returned to the client. This four-layer security protection process is as follows: Figure 2 As shown below, the specific implementation methods of each layer and mechanism are described in turn.

[0026] The rule-based detection layer is the entry point of the entire protection system. It is always executed on every piece of text to be detected. Its core consists of three components: Bloom filter pre-filtering, AC automaton multi-pattern matching, and regular expression-assisted detection.

[0027] Considering that large model input text often contains a variety of characters, including mixed Chinese and English, emojis, Unicode compatibility variants, zero-width characters, and a mix of full-width and half-width characters, and that attackers often circumvent string matching by inserting zero-width spaces (U+200B), zero-width connectors (U+200D), and visually identical but different code-position compatible characters, a character encoding normalization preprocessing submodule is set up before the Bloom filter and AC automaton. This submodule performs Unicode compatibility normalization on the text to be detected (using NFKC normalization to restore full-width letters and compatible characters to their standard equivalent forms), strips zero-width characters and bidirectional control characters (such as U+200B, U+200C, U+200D, U+2060, U+FEFF, U+200E, U+200F, etc.), and performs full-width and half-width unification, case folding, and combination character sequence normalization. Emojis are treated as single markers in the matching to avoid the splitting of proxy pairs, which would lead to mismatches.

[0028] Regarding semantic variant attacks such as homophones, mixed pinyin, and character splitting, the encoding model of the subsequent vector detection layer is responsible for identification and interception.

[0029] Bloom filters are used for fast pre-screening before exact matching in an AC automaton. Their implementation involves initializing an M-bit array with all positions set to 0, and then calculating hashes for all known prohibited and sensitive words using K independent hash functions, setting the corresponding positions to 1. The normalized text to be detected is segmented according to an N-gram syntax (N is usually 2 or 3). The same K hash functions are calculated for each segment. If all K hash mapping positions of a segment are 1, the segment is considered to have a possible violation and triggers a subsequent AC automaton for exact matching. If any position is 0, the segment is considered unlikely to have any known violation, thus skipping the AC automaton matching. The Bloom filter has extremely high space efficiency, enabling fast pre-filtering of large-scale dictionaries with minimal memory usage, significantly reducing unnecessary AC automaton matching. Its bit array size and the number of hash functions are calculated based on the preset false positive rate and the size of the dictionary. The regular expression-assisted detection is executed in parallel with the Bloom filter and the AC automaton branch and is unaffected by the Bloom filter pre-filtering results.

[0030] The AC automaton multi-pattern string matching algorithm is the core data structure for implementing multi-pattern concurrent matching. Its construction process is as follows: All enabled prohibited terms, sensitive terms, and privacy rules are loaded from the database and inserted into the trie one by one. Each trie node contains a child node mapping, a failure pointer, a flag indicating whether it is the term's endpoint, and a list of associated scenario rule identifiers. If a term's endpoint node already exists (i.e., a common prefix exists), the scenario rule identifier of that term is appended to the identifier list of that node.

[0031] Then, a breadth-first search algorithm is used to traverse the trie and calculate the failure pointer for each node. The failure pointer of the root node points to itself, the failure pointers of the root node's direct children point to the root node, and the failure pointers of other nodes point to the trie node corresponding to the longest proper suffix of the string corresponding to that node. In the specific calculation, let the current node be the current node and the node pointed to by the failure pointer of its parent node be the reference node. If the reference node has a child node with the same character as the current node, then the failure pointer of the current node points to that child node. Otherwise, continue to backtrack upwards along the failure pointer of the reference node until the root node.

[0032] During the matching process, the text to be detected is input into the automaton character by character. If the current state node has a child node that is equal to the input character, the process transitions to that child node. Otherwise, the process backtracks along the failure pointer until an ancestor node with a child node containing that character is found or backtracks to the root node. After each state transition, the current node and all nodes along the failure pointer link whose endpoint flags are true are checked. The words corresponding to these nodes are the matched sensitive words.

[0033] The matched terms need to be associated with the current request's business scenario. The system maintains a mapping table between scenarios and rules, and only when the matched term belongs to the activation rule of the current scenario is it considered a valid hit. Then, actions are taken based on the risk level of the hit: if a high-risk hit occurs, the system directly returns to a dangerous state, executes an interception, and does not proceed to subsequent detection layers; if a medium- or low-risk hit occurs, the hit information is recorded and the system proceeds to the next detection layer. Regular expression-assisted detection is used to detect sensitive information with fixed format patterns, such as 11-digit mobile phone numbers, 18-digit ID card numbers with checksums, standard email addresses, etc. It is executed in parallel with AC automaton detection, and an interception is triggered if either detection result is dangerous.

[0034] The rule detection layer outputs a first-level risk score: when a high-risk rule is hit, the risk score is set to 1.0 and the text is blocked in a dangerous state; when a medium-risk or low-risk rule is hit, or no rule is hit, the state is non-dangerous, and the risk score is mapped to the range of 0 to 1 by the severity coefficient of the hit rule and the number of hits according to a preset normalization function. The per-input reliability of this layer is determined by the coverage of the text to be detected relative to the risk vocabulary. The higher the proportion of out-of-vocabulary words in the text to be detected, or the lower the reliability when this layer does not produce any hits or near-hits, the lower the weight of this layer's judgment in subsequent accumulation when the text is suspected of being obfuscated and cannot be covered by the vocabulary. The average execution time of the rule detection layer is less than 5 milliseconds, and it can cover approximately 40% of known sensitive content.

[0035] The vector detection layer is triggered when the rule-based detection layer decides to allow a request. It is mainly used to identify adversarial attacks that circumvent keyword detection through synonym substitution, variant expressions, and mixed pinyin spelling. This layer uses a pre-trained text encoding model to map the text to be detected into a dense vector of fixed dimensions. The vector dimension is typically 256 to 2048, preferably 768 or 1024, depending on the output dimension of the encoding model used.

[0036] This layer stores vectors of known sensitive terms and performs approximate nearest neighbor retrieval based on a vector retrieval engine with a hierarchical navigable small world (HNSW) index. The HNSW index contains a multi-layer graph structure, with sparse nodes in the upper layer for fast location of the search area and dense nodes in the lower layer for fine-grained search. The maximum number of connections for each node is usually an integer between 8 and 64, preferably 16 or 32. The search depth during construction is usually between 100 and 500, preferably 200. The distance metric uses cosine similarity, which is usually implemented by first L2 normalizing the vectors and then calculating the cosine similarity through the inner product to improve retrieval efficiency.

[0037] During retrieval, the text to be detected is encoded into a query vector. The index is then used to retrieve the K most similar known sensitive term vectors to the query vector. K is typically an integer between 3 and 50, preferably between 5 and 25. The cosine similarity between the query vector and each nearest neighbor vector is calculated, and the maximum value is taken as the risk score for the second layer. This layer uses a dual threshold for three-state judgment: if the highest similarity is greater than or equal to the high threshold (typically 0.75 to 0.95, preferably 0.85), it is judged as dangerous and blocked in a dangerous state; if the highest similarity is between the low threshold (typically 0.50 to 0.75, preferably 0.65) and the high threshold, it is judged as suspicious, and its score is passed to the subsequent layers. Safety is determined when the highest similarity is less than a low threshold. To improve the generalization ability to new variants outside the training distribution, the second-layer risk score can also be determined by combining the projection energy of the risk subspace: a set of risk direction vectors or risk prototype vectors obtained from historical positive and negative samples through comparative learning are preset for each risk category to span the risk subspace corresponding to that risk category. The projection energy of the text vector to be detected on each risk subspace is fused with the nearest neighbor similarity retrieval results, and the fused result is taken as the second-layer risk score. This method describes the direction of the risk rather than the distance to a certain enumerated bad word.

[0038] The per-input reliability of this layer is determined by the difference between the highest and second-highest cosine similarity in its search results, or the distance between the highest cosine similarity and the decision threshold. The larger the difference or distance, the higher the reliability; the reliability is low when the similarity distribution is flat or close to the threshold. The average execution time of the vector detection layer is 20 to 50 milliseconds, and the cumulative coverage can reach approximately 65% ​​after combining with the first layer.

[0039] The semantic detection layer is triggered when the vector detection layer determines whether to allow a pass, forming a scalable small model detection platform. It supports concurrent calls to multiple lightweight detection models and tools, and outputs a comprehensive risk score through a weighted fusion strategy. The platform adopts a three-stage architecture: registration, concurrency, and fusion. The model registry maintains a registry that records metadata for each registered small model or detection tool, including model identifier, model file path or tool call interface, input / output format definition, execution weight coefficient, timeout parameters, and start / stop status. Administrators can dynamically register or deregister detection units through the management interface without restarting the service.

[0040] The concurrent scheduling engine retrieves all enabled detection units from the registry when the text to be detected arrives and simultaneously distributes the text to each unit for parallel detection. Each unit executes in an independent thread or process without blocking each other. The maximum concurrency is the smaller of the number of detection units and the number of physical cores of the processor. The platform sets a global timeout (default 80 milliseconds). If any unit times out, its result is marked as timed out, its normalized output is filled with 0, and the weights of the remaining units are renormalized proportionally to keep the sum of the weights at 1, without affecting the execution of other units and the final fusion judgment. Result fusion and output generation are performed after all units have completed or timed out. The platform collects the output scores of each unit and the extracted structured information, and weights the confidence scores of each unit according to preset weights to obtain a comprehensive risk score.

[0041] The platform initially integrates three core detection units: the prompt word injection attack detection unit uses a detection model deployed in a common model exchange format to perform binary classification judgment on the text to be detected and output the injection attack probability, which can identify patterns such as direct injection, indirect injection, and separator attack; the semantic risk classification unit performs probability distribution of each risk category; and the context association analysis unit combines context information such as historical dialogue records, user roles, and business scenarios to perform comprehensive risk assessment and output context risk score.

[0042] The platform reserves standard interfaces to support seamless integration of new detection units such as jailbreak interception models based on prompt words, named entity recognition models, code security scanning models, and multimodal detection models. When integrating a new unit, simply add a configuration entry in the registry center, and it will automatically participate in concurrent detection for all subsequent requests without modifying the platform's core code. Weighted fusion is calculated by multiplying the overall risk score by the product of each unit's weight and its risk confidence level. The sum of each unit's weight is 1, and unit weights are typically between 0.05 and 0.50. A typical initial configuration is: prompt word injection detection weight 0.35, semantic risk classification weight 0.35, and contextual association analysis weight 0.30. The weighting should follow these principles: for deterministic units such as exact matching and signature detection, since a hit is strong evidence but the coverage is narrow, a weight of 0.05 to 0.15 is appropriate; for inference units such as injection detection and risk classification based on model classification, since the coverage is broad but there is uncertainty in a single decision, a weight of 0.10 to 0.25 is appropriate; and for auxiliary units such as context association analysis and entity recognition, a weight of 0.05 to 0.15 is appropriate. When a new unit is added, the administrator selects initial weights according to the above principles, and the platform automatically normalizes them. Alternatively, the system can use the outputs of existing and new units from historical interception logs to train meta-learners, take their coefficients as candidate weights, and fine-tune them on the validation set with a margin of ±0.10 to find the optimal weight. The new unit is switched to online after running in shadow mode for one week to confirm there are no anomalies. The deployed detection model is exported through the model exchange format and loaded and executed by the inference engine. Operator fusion, constant folding, and memory reuse optimizations are performed on the exported model. During inference, a scheduling strategy combining static batch processing and dynamic batch processing is adopted. The number of parameters in the detection model ranges from 1 million to 500 million, and the average time for a single inference does not exceed 100 milliseconds.

[0043] The semantic detection layer uses a dual threshold to perform a three-state judgment on the comprehensive risk score: when the comprehensive score is greater than or equal to the high threshold (typically 0.70 to 0.90, preferably 0.80), it is judged as dangerous and intercepted in a dangerous state; when the comprehensive score is between the low threshold (typically 0.40 to 0.60, preferably 0.50) and the high threshold, it is judged as suspicious, and the score and extracted structured information are passed forward together. The structured information includes a list of risk labels identified by each unit, a list of sensitive entities, attack feature descriptions, and confidence scores; when the comprehensive score is less than the low threshold, it is judged as safe. The per-input reliability of this layer is determined by the variance or prediction entropy of the risk scores output by its concurrent detection units. The reliability is high when all units are consistent and the entropy is low, and low when there is a large divergence between units or when the entropy is high. The average execution time of the semantic detection layer is 30 to 80 milliseconds, and the cumulative coverage after combining with the first three layers can reach approximately 85%.

[0044] If no interception occurs at the rule detection layer, vector detection layer, and semantic detection layer, the process proceeds to cross-layer cumulative decision-making. The cross-layer reliability calibration, dynamic weighting, and dual-criteria upgrade process is as follows: Figure 3 As shown.

[0045] Considering that the output scores of the three layers have different dimensions, the severity heuristic score of the rule layer, the cosine similarity of the vector layer, and the model confidence of the semantic layer are on different dimensions, this invention first performs probability calibration on the scores of each layer: a calibration mapping is preset for each detection layer. The calibration mapping is obtained by fitting the final review result or manual review label of the asynchronous final review layer as the truth value, the risk score of the layer as the input, and the historical detection log. Temperature calibration is used for the semantic detection layer, and ordinal regression or Platt calibration is used for the rule detection layer and the vector detection layer. The historical detection log is divided into a training set and a validation set according to the time sequence. The calibration mapping is fitted on the training set and confirmed on the validation set to avoid time leakage. After calibration, the risk scores of each layer are mapped to calibrated risk probabilities with a unified probability dimension, so that the subsequent weighted summation has probabilistic meaning.

[0046] The present invention further determines the per-input reliability of each detection layer for the current text to be detected. The reliability is derived from the uncertainty signal of the detection layer's own judgment of the current text to be detected. The reliability of the rule detection layer is determined by the coverage of the text relative to the risk word library or the proportion of out-of-vocabulary words. The reliability of the vector detection layer is determined by the difference between the highest cosine similarity and the second highest cosine similarity in the search results or the distance between the highest cosine similarity and the judgment threshold. The reliability of the semantic detection layer is determined by the variance or prediction entropy of the risk scores output by its concurrent detection units.

[0047] Subsequently, the preset base weights of each detection layer are multiplied by the corresponding per-input reliability and normalized to obtain the per-input effective weights. That is, the effective weight of a certain layer is equal to the product of the base weight of that layer and the per-input reliability divided by the sum of the products of all layers. The cross-layer cumulative risk score is obtained by weighting and summing the calibration risk probabilities of each layer with the per-input effective weights. When a layer with a low base weight is highly reliable for the current input and indicates risk, its vote is adaptively amplified and the false alarms are reduced. When a layer is unreliable for the current input, its noise is suppressed and the false alarms are reduced.

[0048] The basic weights adopt an asymmetric increasing distribution from the rule detection layer to the semantic detection layer. That is, the basic weight of the rule detection layer is less than the basic weight of the vector detection layer, and the basic weight of the vector detection layer is less than the basic weight of the semantic detection layer. This reflects the increasing trust level of each layer from strong determinism but narrow coverage to wide coverage but high single uncertainty. Typical values ​​are 0.15 for the rule layer, 0.30 for the vector layer, and 0.55 for the semantic layer. The basic weights, along with the cumulative threshold, divergence threshold, and lower limit, are all obtained through post-event statistical analysis based on the historical detection logs of the deployment environment and are configured independently for different business scenarios, rather than being set a priori.

[0049] This invention uses a combination of independent cumulative risk criteria and inter-layer divergence criteria to determine whether to upgrade to layer 4. The cumulative risk criterion is that the cross-layer cumulative risk score is greater than or equal to a preset cumulative threshold. The cumulative threshold typically ranges from 0.50 to 0.75, with a typical value of 0.65. Its lower bound ensures that an upgrade is triggered only when there is a moderate risk in multi-layer collaboration, so as to avoid single-layer noise pushing up the cumulative score. Its upper bound ensures that the cumulative score still has a fallback capability against covert attacks.

[0050] The interlayer divergence criterion is that the interlayer divergence degree is greater than or equal to a preset divergence threshold and the maximum value of each calibration risk probability is not less than a preset lower limit. The interlayer divergence degree characterizes the dispersion of each layer's calibration risk probability and is calculated based on the weighted variance of each calibration risk probability relative to the cross-layer cumulative risk score. The lower limit is used to suppress unnecessary escalation caused by pure noise.

[0051] Considering that weighted averaging discards information about dispersion, when a novel evasion attack generates a strong signal only at a single detection layer while other layers are deemed safe, the cumulative score after weighted averaging may be lowered below the cumulative threshold. The divergence criterion uses the variance of the same set of calibration scores as a second escalation condition alongside the cumulative criterion, thus specifically restoring coverage for such covert attacks without lowering the overall cumulative threshold (i.e., without sacrificing global false positives and Layer 4 load). If either criterion is met, the request is deemed suspicious and escalated to Layer 4 asynchronous final review; if neither criterion is met, the request is deemed safe. Furthermore, the cumulative threshold can be adaptively adjusted based on the degree of divergence; that is, the effective cumulative threshold obtained by subtracting a non-negative coefficient from the original cumulative threshold and the product of the divergence degree replaces the original cumulative threshold, so that the greater the divergence, the lower the escalation threshold.

[0052] As an example, when all three layers are deemed suspicious but none have reached their respective danger thresholds, assuming the calibration risk probabilities of the three layers are approximately 0.4, 0.55, and 0.6 respectively, and the effective input weights are approximately 0.1, 0.3, and 0.6 after reliability modulation and normalization, the cumulative risk score across layers is approximately 0.565. If the cumulative threshold is 0.65, the cumulative criterion is not valid. However, if the dispersion of the scores of the three layers is large, causing the inter-layer divergence to exceed the divergence threshold and the maximum calibration probability to exceed the lower limit, then the divergence criterion is valid, and the request is still upgraded to the fourth layer, thus avoiding the omissions that may occur if only the weighted average is used.

[0053] As can be seen from the above, this invention constitutes a two-layer defense structure combining single-layer immediate interception of danger and cross-layer comprehensive score-based fallback, and further covers single-layer strong signal-type covert attacks with divergence criteria; this upgrade decision is mathematically indivisible to any single weighted sum. The system also supports automatic selection of threshold configuration based on business scenario characteristics: multiple sets of calibrated and verified threshold templates are pre-set, each template containing complete threshold and weight configurations and descriptions of applicable scenarios. For example, a higher threshold is preferred for general intelligent customer service scenarios to ensure user experience; a lower threshold is appropriately used for zero-tolerance scenarios such as financial customer service to reduce false negatives; a higher threshold is appropriately used for performance-sensitive scenarios such as code generation to improve throughput; and interception is preferred for high-risk scenarios such as content creation. The system automatically matches the closest template from the template library as the initial configuration based on the current business request volume, risk tolerance, and latency constraints. During operation, it can automatically switch to a more suitable template as the request pattern changes (such as a sudden increase in attack traffic or the launch of a new business scenario), without interrupting the service.

[0054] Representative threshold template configurations are shown in Table 1. Those skilled in the art can further calibrate and adjust them in their own business scenarios based on this and the aforementioned post-event statistical analysis method based on historical interception logs.

[0055] Table 1

[0056] The fourth layer, asynchronous final review, is triggered when the cross-layer cumulative decision-making determines something is suspicious. It serves as the final fallback detection layer in the entire protection system. Unlike the synchronous detection of the first three layers, the fourth layer employs an asynchronous architecture to avoid the high latency of large model inference blocking the main request process. Its process is as follows: before sending the text to be inspected to the large model review service, format-preserving sensitive information replacement is performed. This replaces personal identification information in the text with format-preserving placeholders, ensuring that the de-identified text does not contain any sensitive information and can be securely transmitted to the public network. The anonymized text to be detected, the results of the first three layers of detection, the request identifier, and the business scenario are encapsulated into an audit task message and pushed to the message queue system. The message queue system can be a distributed message middleware that supports persistence and multiple consumers, or a memory queue system that is simple to deploy and has lower latency, depending on the system scale and performance requirements. The large model audit service, as a message queue consumer, obtains audit tasks from the queue and calls the security content audit large model to perform deep semantic analysis and intent recognition, generating audit results that include risk category, risk level, risk confidence, and handling suggestions. After the review is completed, the results and details are written to the detection result storage module, and an association mapping between the review result and the original request is established based on the request identifier. Then, the detection result query service is notified through a callback mechanism. The callback methods include real-time push for scenarios with high real-time requirements and polling of the client's periodic query interface for review results. The inference latency of the fourth-layer model is more than 200 milliseconds, but because the asynchronous architecture does not block the main request processing flow, the user can obtain the final review result through the callback mechanism after receiving the initial response.

[0057] The review results of the fourth-layer large model are not only used for the current request processing, but also used as training data for knowledge distillation from the large model to the small model. Samples that the large model judges to be dangerous are automatically labeled and added to the training sample set for incremental training of the detection model in the semantic detection layer, thereby continuously improving the accuracy of the small model and reducing the proportion of requests that need to enter the fourth-layer review.

[0058] Specifically, the large-scale security content review model outputs the probability distribution of each risk category from the input text as a soft label. The distillation loss adopts a combination of cross-entropy and temperature softening, with the temperature coefficient typically ranging from 3 to 10, preferably 5. The total loss is the weighted sum of the distillation loss and the hard label loss of the small model relative to the true label, weighted by the distillation weight coefficient, typically ranging from 0.3 to 0.8, preferably 0.5. The training adopts a daily incremental fine-tuning method, using the samples accumulated on that day and labeled by the large model as the training set to perform 1 to 5 rounds of iterative training on the small model. The learning rate typically ranges from 10 to the power of -5 to 10 to the power of -3, preferably 5 times 10 to the power of -5.

[0059] This invention further modifies the above distillation into selective distillation oriented towards the decision boundary: from the samples that have passed the final review of the fourth layer, the distillation weight of each sample is determined according to the difference between the risk probability output by the large model and the calibration risk probability output by the semantic detection layer, or according to the per-input reliability of the semantic detection layer for the sample. This allows boundary samples with large differences between the teacher's and student's judgments or high uncertainty in the student's judgments to obtain higher distillation weights. The small model is then incrementally trained after the distillation loss is weighted by the distillation weights, thereby concentrating the training budget on the weak areas of the students and increasing the pressure drop speed of the unit labeled sample to the fourth layer upgrade ratio. Furthermore, the calibration mappings of each layer are periodically refitted and the basic weights of each layer are re-estimated using the final result of the fourth layer as the truth value. This ensures that the closed loop not only updates the parameters of the small model but also updates the fusion decision mechanism itself upon which the upgrade judgment is based. The aforementioned selective distillation and fusion self-calibration closed loop is as follows: Figure 4 As shown, this ensures the long-term reliability of the cumulative score despite attack distribution drift.

[0060] Because daily intercepted samples are naturally biased towards known attack patterns, long-term incremental training may lead to a degradation in the small model's ability to distinguish historical attack patterns. Furthermore, the distribution bias of intercepted logs may cause the small model to underfit low-frequency attack patterns. Therefore, this invention employs a catastrophic forgetting constraint mechanism in incremental training. This mechanism includes experience replay, where daily training samples consist of a mixture of newly added samples (typically accounting for 0.6 to 0.8%) and historical sample replays. Historical sample replays are sampled from a long-term sample pool that uses a circular buffer structure and retains representative samples from the past 90 to 180 days, using an attack category balancing and time decay weighting strategy to ensure that all historical attack categories have a certain proportion in each round of training.

[0061] The mechanism also includes elastic weight solidification, which introduces a regularization term in addition to distillation loss. This regularization term is the sum of half the regularization strength coefficient and the product of the diagonal elements of the Fisher information matrix of each parameter and the square of the deviation of the parameter from the previous stable version. The diagonal elements of the Fisher information matrix measure the importance of the parameter to the historical task and are updated on the historical guard set every 7 days. The regularization strength coefficient is typically between 100 and 1000.

[0062] The mechanism also includes maintaining a set of historical safeguard evaluation sets that are completely isolated from the training data and are balanced by attack category (at least 200 samples per category, rolled over every six months). New models must be validated on this safeguard set before deployment. If the recall rate of any historical attack category drops more than a preset threshold (typically 2%) relative to the previous version, deployment will be rejected and a rollback will be triggered.

[0063] The mechanism also includes performing distribution drift detection on the intercepted samples of the day and the historical sample pool (using the population stability index or the difference of the maximum mean). When the drift index exceeds the preset threshold (e.g., the population stability index exceeds 0.25), it is considered that a significant distribution shift has occurred and the distillation training for the day is automatically suspended, the current version of the online model is kept unchanged, and the operation and maintenance personnel are notified to review the quality of the interception logs.

[0064] The online model is deployed using versioning, retaining at least the three most recent versions. When performance degradation is detected after a new version is launched (e.g., the 7-day moving average of the false negative rate exceeds the warning threshold), an atomic pointer switch is used to roll back to the previous stable version. The rollback time is less than 1 second and does not interrupt online services. The new model is only deployed as an online model via hot replacement when all four conditions are met: the F1 score on the validation set is not lower than that of the original model, the recall rate is not lower than that of the original model, the recall rate degradation of any category on the historical guard set does not exceed 2%, and the inference latency increase does not exceed 14%.

[0065] Before invoking the public network security content review model and before allowing requests to the target model service, sensitive information in the text needs to be replaced with format-preserving replacements. This ensures that the anonymized text retains its original format characteristics without affecting the model's understanding of the text structure. This invention identifies seven types of sensitive information: 11-digit mobile phone numbers starting with 1; 18-digit ID card numbers consisting of 17 digits plus a 1-digit checksum (the checksum can be numbers or letters); email addresses in username-domain format; bank card numbers of 16 to 19 digits; physical addresses containing elements such as province, city, district, and street; names of 2 to 4 Chinese characters; and license plate numbers consisting of abbreviated province names plus alphanumeric combinations.

[0066] This invention provides seven desensitization operations: masking by replacing part of the characters with specific characters, replacing the entire sensitive information with a preset placeholder, truncating only the prefix or suffix, encrypting with a symmetric encryption algorithm while retaining the ciphertext format, calculating and retaining a hash with a fixed length representation using a hash algorithm, generalizing specific information into a broader category, and randomizing to generate random fake data in the same format.

[0067] The replacement strategies for various sensitive information are as follows: For mobile phone numbers, the first 3 and last 4 digits are retained, and the middle part is masked with asterisks; for ID card numbers, the first 6 and last 4 digits are retained, and the middle part is masked with asterisks; for bank card numbers, the first 4 and last 4 digits are retained, and the middle part is masked with asterisks; for email addresses, the first character before the @ symbol and the complete domain name after the @ symbol are retained, and the middle part is masked with asterisks; for physical addresses, the province / city level information is retained, and the district and lower levels are replaced with asterisks; for names, the surname is retained, and the first part is masked with asterisks; for license plate numbers, the province abbreviation and first letter are retained, and the subsequent characters are masked with asterisks.

[0068] The replacement needs to meet four specific requirements in the large model scenario and is designed accordingly: First, the desensitized text is still used as input to the large model. Therefore, the desensitization method cannot destroy the semantic structure and syntactic coherence of the text. This invention uses placeholders of the same length and structure (such as desensitizing the name by keeping the surname and adding an asterisk, desensitizing the address by keeping the province and city and adding an asterisk, and keeping the prefix and suffix of numeric information and adding an asterisk mask in the middle) so that the large model can still recognize the semantic role of the position as a person's name, address or a certain type of number. Secondly, placeholders should not be misread as real content or structured tags by the large model. Therefore, the default format is to maintain the same visual format as the original data to keep placeholders rather than eye-catching placeholders. When placeholders are necessary, use ordinary Chinese asterisks or ellipses that conform to reading habits, and avoid using symbols such as angle brackets, curly braces, and square brackets that may be parsed as structured tags. Placeholders do not participate in the control of the tag character set. Thirdly, the length of the replaced placeholder string is strictly controlled to be no greater than the length of the original sensitive entity, and the number of characters after generalization is also no greater than the number of characters in the original string, so as to coordinate with the limited context window length of the large model and avoid the request being truncated. Fourthly, it supports session-level consistent restoration in multi-turn dialogue scenarios of the large model.

[0069] To support authorized callers in restoring and ensuring consistency across multiple rounds, this invention stores the mapping relationship between the original sensitive information and the de-identified value in a key-value or hash data structure of the cache database, using the session identifier as the group key. The key space is shaped like a de-identified mapping key identified by the session identifier, with fields serving as de-identification placeholders and values ​​as the original plaintext. Multiple de-identifications of the same original value within the same session return idempotent placeholders to ensure consistency across rounds. The mapping relationship is set with an expiration time aligned with the session lifecycle (typically 24 hours), and is automatically deleted after expiration to avoid long-term data retention.

[0070] When multiple sensitive information entries exist in the text, they are replaced in reverse order from back to front. Because replacement changes the text length, reverse replacement ensures that the replacement position of each sensitive information entry is not affected by previous replacement operations. Only authorized downstream services can read the mapping from the cache database and perform restoration using the session identifier and service identity credentials. The public network security content review big model service and external target big model services will always only see the de-identified placeholder string.

[0071] This invention further sets up sensitive information leakage interception on the output side: after the target large model service returns a response and before returning a response to the client, reverse detection is performed on the response. For the placeholder string in the response that corresponds to a certain de-identified placeholder in the session-level mapping relationship, only authorized callers are restored to the original value according to the mapping. Sensitive information that appears in the response but does not have a corresponding original value in the session-level mapping relationship is judged as unauthorized leakage and is intercepted or re-de-identified and recorded in the audit log, thereby capturing personal identity information reconstructed or fabricated by the large model based on multiple rounds of context. Furthermore, any text sent to the security content review big model and target big model service is verified based on session-level mapping relationships to ensure it does not contain any original sensitive information values ​​before replacement. This constitutes a two-way end-to-end fence combining input-side desensitization and output-side reverse verification. The data flow of this two-way desensitization fence is as follows: Figure 5 As shown.

[0072] To address evolving attack methods, this invention designs four dynamic learning mechanisms: hot rule base updates, incremental vector library training, daily incremental model learning, and automatic storage of intercepted samples. The hot rule base updates employ a double-buffering mechanism, maintaining two instances: an online rule base and a shadow rule base. When an update is needed, a new AC automaton trie, Bloom filter, and regular expression rule set are constructed on the shadow rule base. After construction, atomic operations are used to compare and exchange references in the online rule base to point to the shadow rule base, while simultaneously converting the original online rule base to the shadow rule base. This ensures that the rule base will not be empty or inconsistent during the switching process, and the entire switching process takes less than 1 millisecond.

[0073] In terms of concurrency safety, the global rule engine pointer is held by atomic references. When online request processing begins, a local cached pointer snapshot is used to complete the matching throughout the entire request lifecycle. During the switch, the new pointer is replaced by comparison and atomic setting. The old instance is recycled through reference counting or read-copy update mode after all requests holding its reference are completed. Requests in progress during the switch continue to use the old rules without being blocked, and new requests after the switch use the new rules. For a rule base of millions, the typical time taken for a single complete construction is 2 to 5 seconds for the trie, 1 to 3 seconds for the failed pointer, 0.5 to 1 second for the Bloom bit array, and 1 to 2 seconds for the regular expression cache, totaling 5 to 12 seconds, all executed in the background.

[0074] The incremental training of the vector library adopts a shadow indexing mechanism, which constructs an independent shadow index that runs in parallel with the online index. New sensitive terms are first vectorized and encoded in the shadow index and an index is built. When the number of new terms in the shadow index reaches a preset threshold or the incremental period is reached, it is merged with the online index. After the merge is completed, the online index reference is pointed to the merged index through atomic switching. If the switching fails, it automatically rolls back to the original index. The merge is executed asynchronously in the background and does not affect online detection.

[0075] Since HNSW indexes cannot be simply merged (nearest neighbor relationships across subgraphs will be lost), this invention adopts an incremental insertion scheme. Each vector in the shadow index is inserted into the online index one by one according to the standard algorithm, and local rebalancing is performed on the affected layers. Edges exceeding the maximum number of neighbors are deleted. When the size of the shadow index exceeds 20% of the online index, a full reconstruction is performed.

[0076] The model performs daily incremental learning by collecting intercepted samples deemed dangerous by any layer the previous day. These samples are then automatically regularized, including removing Hypertext Markup Language tags, denoising characters and invisible characters, converting them to Unicode and performing NFKC normalization, standardizing punctuation for both Chinese and English punctuation, truncating excessively long samples by sentence boundaries, and deduplicating samples with a minimum hash similarity greater than or equal to 0.95. The regularized samples are then used to incrementally train the small model. After offline verification that its performance is no less than that of the original model, it is then hot-replaced and deployed.

[0077] The automatic sample interception mechanism automatically extracts the request text, matching terms or rules, detection results and risk categories when a request is judged as dangerous by any layer, and stores them in the training dataset and labeled data source layer. It also periodically deduplicates and performs quality checks on the training dataset, which serves as the data source for rule base updates, incremental training of vector library and incremental model learning.

[0078] The specific algorithm for automatically generating candidate rules from intercepted samples is as follows: samples are clustered based on density according to text embedding vectors to obtain attack pattern clusters. Within each cluster, samples are extracted using the longest common substring or suffix array to extract common substring patterns. The common substring is used as the skeleton, and the changing parts are generalized to regular expression metacharacters of number, whitespace, and character classes to generate candidate regular expressions. Candidate regular expressions are verified by the historical guard set and pass the initial screening if the false alarm rate is less than 1%. They are then manually reviewed and confirmed before being added to the rule base and then hot-updated and launched through the rule base.

[0079] The evaluation set and training set are strictly isolated by time window (the evaluation set only uses samples after the training deadline). Sampling is performed in layers according to attack category or business scenario. Each month, a manual review team randomly checks 5% of the samples to verify the label quality. The evaluation set and training process are isolated through file system permissions to avoid training data contamination.

[0080] From a system perspective, the system of this invention includes a request access module, a rule detection module, a vector detection module, a semantic detection module, a cross-layer cumulative decision module, an asynchronous review module, a sensitive information replacement module, a result assembly module, and a dynamic learning module. The request access module receives client requests, performs API authentication and business scenario matching, and extracts the text to be detected and its context information; the rule detection module, vector detection module, and semantic detection module respectively implement the aforementioned three-layer synchronous detection and output risk scores and status or execute interception. The cross-layer cumulative decision module is connected to three detection modules and an asynchronous review module. It calibrates the risk scores output by the three modules, weights the input reliability to obtain the cross-layer cumulative risk score, and makes a comprehensive judgment based on the cumulative criterion and the inter-layer divergence criterion. Requests deemed suspicious are routed to the asynchronous review module, and requests deemed safe are routed to the result assembly module. When the asynchronous review module determines something to be suspicious, it asynchronously submits the text to be detected, context information, and structured information to the security content review big model service through a message queue for the fourth layer final review. The sensitive information replacement module performs format-preserving sensitive information replacement and establishes a session-level mapping relationship before allowing access to the target large model service. This is used for authorization restoration and output-side verification. The result assembly module receives the detection results and final review results from each layer and assembles them. Based on the final judgment result, it performs either release or interception. The dynamic learning module connects to the three detection modules respectively and performs hot updates of the rule base, incremental training of the vector base, and incremental learning of the model.

[0081] The system of this invention supports multiple deployment methods, including independent deployment as an independent service process integrated with the large model service via Hypertext Transfer Protocol or its secure version, embedded deployment as a network application module that intercepts and detects requests and responses via filters or interceptors, and sidecar deployment in a container orchestration environment where it shares the network namespace with the large model service container as a sidecar container.

[0082] The general technical components described in this embodiment can be implemented by those skilled in the art using existing open-source or commercial products. As a non-limiting specific implementation example, the model inference engine can be an open-source cross-platform machine learning model inference engine, which loads pre-trained model files through its inference session interface, configures the central processing unit or graphics processor to execute the provider to perform forward inference, the general model exchange format can be an open neural network exchange format, and exports the trained model as a corresponding file, the vector retrieval engine can be a vector database that supports HNSW indexes, and builds indexes and performs nearest neighbor retrieval through its index and retrieval commands, the cache database can be a memory database that supports string and hash data structures and sets the lifespan with expiration commands, the distributed message middleware can be a message system that supports publish-subscribe mode, the network application framework can be an application framework that supports filters or interceptors, the object-relational mapping component can be a corresponding persistence layer framework, and the relational database can be a corresponding relational database management system. The above products are merely examples; any component with equivalent functionality can be used to implement the methods and systems described in this invention. All detection results are recorded in the protection log, including request identifier, user information, scenario information, detection results at each layer, matched terms or rules, handling actions and processing time, for use in security auditing, attack analysis, and model training data collection.

[0083] This invention relates to multiple parameters that need to be calibrated, including the high and low thresholds of the vector detection layer, the fusion weights of each detection unit in the semantic detection layer, the high and low thresholds of the semantic detection layer, the temperature, weights, and learning rate of the fourth layer distillation, as well as the cross-layer cumulative weights and cumulative thresholds, the divergence thresholds, and the lower limit. All parameters are calibrated using a unified post-event statistical analysis method based on historical interception logs.

[0084] The methodology is as follows: Using the outputs of each layer in the current network interception logs, the final review conclusion of the fourth layer, and the manual review label as data sources, and taking the fourth layer final review or manual review as the truth value, the data is divided into training, validation, and test sets (typically in a 7:1:2 ratio) according to time sequence to avoid time leakage. A grid search or Bayesian optimization is performed on the parameter space. On the validation set, the optimal and stable parameters are selected with the goal of minimizing the false negative rate and not increasing the false positive rate. On the test set, the generalization metric is confirmed, and sensitivity curves for each parameter are plotted near the optimal parameters, along with comparison curves for enabling and disabling cross-layer accumulation and divergence criteria. The above calibration is performed for different business scenarios to obtain scenario-specific parameters. Recalibration is required when the proportion of new attacks in the interception logs exceeds 10%, one of the three-layer detection components undergoes a major upgrade, the rule base or vector base undergoes large-scale reconstruction, or the fixed period expires.

[0085] The following exemplary test data under a typical test environment is provided to illustrate the beneficial effects of the present invention. Those skilled in the art will understand that actual performance indicators may vary due to differences in hardware configuration, network environment, and test dataset. This specification also discloses experimental methodologies to enable those skilled in the art to reproduce the results in their own deployment environment. The exemplary test environment is a 32-core server processor, 64 gigabytes of memory, coupled with a vector retrieval engine, a network application framework, and a model inference engine. The exemplary test dataset consists of 10,000 samples, mixed with real-world attack samples, publicly available attack dataset samples, red team-generated adversarial samples, and samples outside the training distribution in typical proportions of approximately 30%, 30%, 25%, and 15%. It covers categories such as prompt injection, adversarial samples, privacy data leakage, control tag hijacking, multi-round context hijacking, and hybrid attacks. The comparison schemes include single-layer schemes such as pure rules, pure vectors, pure small models, and pure large models, as well as multi-layer schemes such as simple concatenation of rule vector small models and synchronous concatenation with large models. Public metrics of representative industry schemes are included as external references. The experiment was repeated at least 5 times, and the mean and standard deviation were taken. The 95% confidence interval was calculated for the key indicators. The McNimar test or paired t test was used to evaluate the significance of paired samples (statistical significance was defined as less than 0.01). The adversarial robustness test included three types: white-box gradient attack, black-box migration attack and automated red team tools.

[0086] On a balanced dataset where security samples and attack samples each account for half, exemplary results for each scheme are shown in Table 2. The 4-layer scheme of this invention completes the detection of approximately 83.6% of requests at the rule detection layer, and only about 0.7% of requests need to enter the 4th layer asynchronous review. The industry multi-layer baseline rows in Table 2 are taken from the indicator range contained in external public information, and the false positive rate is expressed as N / A because it is not disclosed in the public information.

[0087] Table 2

[0088] Considering that the proportion of attack samples in real production environments is usually much lower than that of balanced datasets, in order to evaluate the robustness of this invention under real business distribution, the test set was resampled and various indicators were measured under different attack sample proportions. The results are shown in Table 3: Table 3

[0089] Five independent replicate experiments were conducted on the core indicators (each using different random seeds and test sets for sampling), and the mean, standard deviation, and 95% confidence interval were calculated. The results are shown in Table 4. The confidence intervals mentioned above do not overlap with the confidence intervals of the pure large model scheme and the simple cascade scheme. The significance level is better than 0.1% under the paired McNimar test, indicating that the performance improvement of the present invention is statistically significant.

[0090] Table 4

[0091] It should be noted that the present invention claims to protect the above-mentioned experimental methodology and the detection and upgrade mechanism based on cross-layer reliability fusion, rather than any specific set of performance values.

[0092] The above-described specific embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Various modifications, substitutions, and improvements made by those skilled in the art to the technical solutions of the present invention based on the provided textual description and drawings, without departing from the design concept and spirit of the present invention, should all fall within the scope of protection of the present invention. The scope of protection of the present invention is determined by the claims.

Claims

1. A large-scale model security detection method based on cross-layer reliability fusion, characterized in that, include: The system receives large model interaction requests and extracts the text to be detected. The text is then sequentially input into multiple cascaded detection layers. Each detection layer includes at least a rule-based detection layer, a vector-based detection layer, and a semantic detection layer based on a lightweight detection model. Each detection layer outputs a normalized risk score. And in The large model interaction request is blocked when the danger threshold of the detection layer is exceeded. If no detection layer intercepts the request, determine whether to escalate the large model interaction request to the asynchronous final review layer based on the security content review large model as follows: Each The calibration risk probability is transformed into a uniform dimension through a preset calibration mapping. ; The input-wise reliability of each detection layer is derived from the uncertainty signal of the current text to be detected. , preset basic weights and Multiply and normalize to obtain the input-wise effective weights ,according to Calculate the cross-layer cumulative risk score and categorize it according to the calibrated risk probability. The inter-layer divergence degree D is calculated relative to the dispersion of the cross-layer cumulative risk score; when Greater than or equal to the preset cumulative threshold Or when the interlayer divergence degree D is greater than or equal to a preset divergence threshold And each The maximum value in the range is not less than the preset lower limit. If the request is deemed suspicious, the interaction request with the large model is considered suspicious; otherwise, it is considered safe.

2. The large-model security detection method based on cross-layer reliability fusion according to claim 1, characterized in that, The calibration mapping is obtained by fitting historical detection logs with the final review result or manual review label of the asynchronous final review layer as the truth value. Temperature calibration is used for the semantic detection layer, and order-preserving regression or Platt calibration is used for the rule detection layer and the vector detection layer. The historical detection logs are divided into training set and validation set according to time sequence to avoid time leakage. The interlayer divergence degree D represents the degree of dispersion of the calibration risk probability of each detection layer, and is calculated based on the weighted variance of each calibration risk probability relative to the cross-layer cumulative risk score.

3. The large-model security detection method based on cross-layer reliability fusion according to claim 1, characterized in that, The determination of the per-input reliability includes: the reliability of the rule detection layer is determined by the coverage of the text to be detected relative to the risk word library. The higher the proportion of out-of-vocabulary words in the text to be detected or the lower the reliability of the rule detection layer when the rule detection layer does not produce any hits or near hits. The reliability of the vector detection layer is determined by the difference between the highest and second-highest cosine similarity in its Top-K search results, or the distance between the highest cosine similarity and the decision threshold. The greater the difference between the highest and second-highest cosine similarity, or the greater the distance between the highest cosine similarity and the decision threshold, the higher the reliability. The reliability of the semantic detection layer is determined by the variance or prediction entropy of the risk scores output by the multiple detection units that are called concurrently. The larger the variance or prediction entropy, the lower the reliability. The cumulative threshold is adaptively adjusted according to the interlayer divergence degree, that is, based on The alternative to the cumulative risk criterion ,in A non-negative coefficient is preset so that the greater the degree of divergence between the layers, the lower the threshold for upgrading the asynchronous final review layer.

4. The large-model security detection method based on cross-layer reliability fusion according to claim 1, characterized in that, The basic weights of each detection layer An asymmetric increasing distribution is adopted from the rule detection layer to the semantic detection layer, that is, the base weight of the rule detection layer is less than the base weight of the vector detection layer, and the base weight of the vector detection layer is less than the base weight of the semantic detection layer. The basic weight The cumulative threshold The divergence threshold and the lower limit Historical detection logs based on the deployment environment are obtained through post-event statistical analysis and are configured independently for different business scenarios.

5. The large-model security detection method based on cross-layer reliability fusion according to claim 1, characterized in that, The method also includes selective knowledge distillation oriented towards decision boundaries, performed from the large-scale secure content moderation model to the lightweight detection model of the semantic detection layer: From the samples that have passed the final review by the asynchronous final review layer, the distillation weight of each sample is determined by the difference between the risk probability output by the security content review big model and the calibration risk probability output by the semantic detection layer, or by the per-input reliability of the semantic detection layer for that sample. Samples with a larger difference in calibration risk probability or a lower per-input reliability are given a higher distillation weight. The distillation loss is then weighted by the distillation weight and incrementally trained on the lightweight detection model. Furthermore, the calibration mapping is periodically refitted and the basic weights are re-estimated using the final review result of the asynchronous final review layer as the truth value. The decision-making mechanism on which the update and upgrade judgment are based will be updated.

6. The large-model security detection method based on cross-layer reliability fusion according to claim 5, characterized in that, The incremental training employs a catastrophic forgetting constraint mechanism, which includes at least one of the following: Experience replay that samples from the historical sample pool and mixes it with the samples of the day, using attack category equalization and time decay weighting; In addition to distillation loss, an elastic weight solidification regularization term based on the Fisher information matrix of the previous stable version parameters is introduced; Maintain a historical guardian evaluation set that is isolated from training data, and refuse to deploy and roll back to the previous stable version when the recall rate of the new model on any historical attack category on the historical guardian evaluation set drops by more than a preset threshold relative to the previous version. Distribution drift detection is performed on the current day's samples and the historical sample pool, and the incremental training is paused when the drift index exceeds a preset threshold.

7. The large-model security detection method based on cross-layer reliability fusion according to claim 1, characterized in that, Before sending the text to be detected to the security content review big model, and before allowing the big model interaction request to be passed to the target big model service, perform format-preserving sensitive information replacement on the text to be detected: Sensitive information entities in the text to be detected are identified and a replacement operation is performed. The length of the replacement placeholder string is no greater than the length of the original sensitive entity and does not use symbols that will be parsed as structured tags by the large model. The original value before replacement and the desensitized value after replacement are stored in the cache database with the session identifier as the group key to form a session-level mapping relationship. Multiple replacements of the same original value within the same session return idempotent placeholders. The session-level mapping relationship is set with an expiration time aligned with the session lifecycle.

8. The large-model security detection method based on cross-layer reliability fusion according to claim 7, characterized in that, It also includes blocking the leakage of sensitive information on the output side: After the target large model service returns a response but before returning a response to the client, reverse detection is performed on the response. For any placeholder string in the response that corresponds to any de-identified placeholder in the session-level mapping relationship, the original value is restored only to the authorized caller based on the session-level mapping relationship. For sensitive information in the response that does not have a corresponding original value in the session-level mapping relationship, it is determined to be unauthorized disclosure and is intercepted or re-identified, and the audit log is recorded. Furthermore, for any text sent to the security content review big model and the target big model service, it is verified according to the session-level mapping relationship that it does not contain any original sensitive information values ​​before replacement.

9. The large-model security detection method based on cross-layer reliability fusion according to claim 1, characterized in that, The risk score of the vector detection layer is also determined by combining the risk subspace projection energy: For each risk category, a set of risk direction vectors or risk prototype vectors obtained through comparative learning are preset to span the risk subspace corresponding to that risk category. The projection energy of the vector of the text to be detected on each risk subspace is fused with the similarity retrieval result of the vector detection layer, and the fusion result is taken as the risk score of the vector detection layer.

10. A large model security testing system based on cross-layer reliability fusion, used to implement the large model security testing method based on cross-layer reliability fusion as described in any one of claims 1-9, characterized in that, include: The request access module is used to receive large model interaction requests and extract the text to be detected; Multiple cascaded detection modules include at least a rule detection module based on keyword and pattern matching, a vector detection module based on vector semantic similarity retrieval, and a semantic detection module based on a lightweight detection model. Each detection module outputs a normalized risk score and outputs a dangerous state to block the request when the risk score exceeds a dangerous threshold. The asynchronous final review module is used to perform deep semantic analysis and intent recognition on upgrade requests based on the large-scale security content review model. A cross-layer cumulative decision module, which is connected to the rule detection module, vector detection module, semantic detection module, and asynchronous final review module, and configured as follows: The risk scores output by each detection module are transformed into a calibration risk probability with a unified probability dimension through a calibration mapping that takes the final review result of the asynchronous final review module or the manual review label as the true value and is fitted based on historical logs. The per-input reliability of each detection module is derived from the uncertainty signal of the judgment of the current text to be detected by each detection module. The per-input reliability is used to modulate and normalize the preset basic weights of each detection module to obtain the per-input effective weights, and the cross-layer cumulative risk score is calculated accordingly. When the cross-layer cumulative risk score is greater than or equal to a preset cumulative threshold, or when the inter-layer divergence is greater than or equal to a preset divergence threshold and the maximum value among the calibration risk probabilities is not less than a preset lower limit, the request is routed to the asynchronous final review module; otherwise, the request is deemed safe. The inter-layer divergence is calculated based on the weighted variance of each calibration risk probability relative to the cross-layer cumulative risk score.

Images