A large language model content security filtering system and method thereof

Abstract

The application relates to the field of artificial intelligence security technology and discloses a large language model content security filtering system and method, which comprises an input protection module, a controllable generation module and an output protection module.The input protection module extracts features of a user request based on a small language model and performs binary classification through an adaptive linear detection mechanism on an optimal hidden state.The controllable generation module adjusts a latent space vector or a probability distribution of a word element to guide the model to generate a safe output.The output protection module detects and purifies unsafe responses with low delay and high detection accuracy, and the output protection module can be continuously evolved to detect emerging harmful responses.The offline optimization module is used for continuously recording conversation histories and detection results of each protection module.Through adaptive linear detection of an optimal hidden state based on a small language model and an evolving multi-expert architecture, low delay, strong detection capability, evolution and unification of emerging attack modes are realized.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence security technology, specifically to a large language model content security filtering system and method. Background Technology

[0002] Large Language Models (LLMs) have rapidly gained popularity and application in numerous fields such as search, finance, office work, security, education, gaming, e-commerce, and social media, demonstrating powerful language understanding, reasoning, and generation capabilities. However, with the widespread use of LLMs, the security risks they face are becoming increasingly severe. Some users exploit malicious prompts to overwrite the original instructions of LLMs to achieve prompt injection attacks. Among these, jailbreak attacks, as a typical type of prompt injection attack, induce the model to leak sensitive information from training, generate content that violates model usage guidelines, or perform other unexpected operations. These attacks are extremely dangerous and have been listed by OWASP as the number one security threat to LLMs.

[0003] To prevent malicious use, large language models are typically equipped with security mechanisms. Current mechanisms mainly employ three approaches: rule-based systems use predefined patterns and keyword filtering for protection; model-based methods utilize traditional classifiers or neural networks for threat detection; and LLM-based systems leverage the reasoning capabilities of large language models for complex analysis. However, all these methods face a fundamental trade-off between computational efficiency and detection accuracy: rule-based systems lack adaptability to new attacks and struggle to cope with rapidly evolving adversarial attack methods; model-based systems require extensive retraining for new threats, resulting in insufficient update timeliness; and LLM-based methods incur high computational overhead, with real-time monitoring demanding significant computing resources, making high-concurrency, low-latency deployment difficult in practice. Most critically, existing methods lack a systematic mechanism to continuously adapt to the evolving adversarial environment without incurring huge computational costs or requiring human intervention, leading to static, rigid security protection that relies on manual iteration. Summary of the Invention

[0004] The present invention aims to provide a content security filtering system and method for large language models, in order to solve the problem that existing large language model security protection mechanisms have a fundamental trade-off between computational efficiency and detection accuracy, and lack a systematic mechanism to continuously adapt to the ever-evolving adversarial environment without incurring huge computational costs or requiring human intervention.

[0005] To solve the above problems, the present invention adopts the following technical solution: Option 1: A content security filtering system for large language models, comprising: An input protection module, configured on the LLM server, receives user requests, extracts features from the user requests based on a small language model, performs binary classification using an adaptive linear probing mechanism on the optimal hidden state, labels prompt words as safe or unsafe, and sends unsafe prompt words to a multi-label classification module for violation type marking. Internally, the input protection module employs an evolutionary multi-expert architecture, using a routing module to assign user prompt words to the most suitable expert classifier. Each expert classifier focuses on a specific input type and continuously evolves to detect emerging harmful prompts and new attack patterns. The controllable generation module, configured on the LLM server, is used to monitor abnormal signals inside the LLM model in real time during the response generation process. By adjusting the probability distribution of the latent space vector or generated lexical units, it guides the model to generate safe output. An output protection module, configured on the LLM server, serves as the final barrier of the protection system. It detects and eliminates unsafe responses with low latency and high detection accuracy. Furthermore, the output protection module can continuously evolve to detect emerging harmful responses. The offline optimization module, configured on the LLM server, is used to continuously record the dialogue history and the detection results of each protection module. It uses automated or semi-automated methods to evaluate the performance of the protection modules, builds a model optimization dataset, and optimizes the defense modules by adjusting model parameters, improving module architecture, or updating system memory. When injecting new knowledge, it ensures that the performance of the protection system remains stable for previous tasks.

[0006] Beneficial effects: This invention constructs a data-driven closed-loop feedback mechanism for recording, evaluating, optimizing, and deploying through three layers of protection in the online inference stage and closed-loop feedback in the offline optimization stage. This enables the system to continuously adapt to emerging jailbreak attack patterns without interrupting online services, achieving a paradigm shift from static defense to autonomous intelligent evolution.

[0007] This invention achieves a unified approach by combining low latency, strong detection capability, and adaptive evolution against emerging attack patterns through optimal hidden state adaptive linear detection based on a small language model and an evolutionary multi-expert architecture.

[0008] Furthermore, the input protection module includes: A binary classification component is used to label prompts as safe or unsafe. A multi-tag classification component is used to mark unsafe warning words as violations. Both the binary classification component and the multi-label classification component employ an expert hybrid structure to route user prompts to the most suitable classifier.

[0009] Beneficial effects: By using a cascaded architecture of binary classification and multi-label classification, fine-grained identification of violation types can be achieved while ensuring low latency, meeting the needs of enterprise-level applications for hierarchical control of content security.

[0010] Furthermore, the adaptive linear detection mechanism on the optimal hidden state includes: Obtain feature representations of each layer of a pre-trained small language model Where B is the batch size, T is the sequence length, and d is the feature dimension; The feature representation is subjected to mean pooling along the sequence dimension to obtain the pooled layer feature vector. ; Train a classifier for each layer to perform binary classification of harmful content, and test the classifier on the validation set. Select the hidden layer with the highest F1 score as the optimal hidden layer. The position of the optimal hidden layer is dynamically adjusted during the expert evolution process.

[0011] Beneficial effects: By dynamically selecting the intermediate layer with the highest security information concentration within the large language model for detection during inference, deep semantic accuracy is obtained at the cost of shallow computation, breaking the technical bias that accuracy is positively correlated with computing power. This allows small language models to outperform general large models in security detection tasks with only a small number of training samples.

[0012] Furthermore, the evolved multi-expert architecture includes: Multiple expert classifiers, each expert being a different linear classifier or a shallow MLP; The routing module classifies based on a small language model and outputs the probability for each expert model. If the confidence level is greater than a preset threshold, the prompt word is sent to the expert model with the highest probability value; if it does not exceed the threshold, all expert models are integrated for classification, and their results are averaged by voting. The expert classifier triggers evolution when the defense accuracy exceeds the threshold. The evolution priority is as follows: fine-tuning only each expert layer, fine-tuning the pre-trained model that obtains feature representations and each expert layer and re-finding the best hidden layer, and adjusting the router and expert model classification.

[0013] Beneficial effects: This invention does not require a large amount of memory when loading multiple experts simultaneously, and through specialized division of labor and a gradual evolution strategy, it can quickly adapt to new attack modes without expanding the model size, thus solving the dilemma of determining the number of experts in traditional expert hybrid systems.

[0014] Furthermore, the controllable generation module includes: An abnormal signal monitoring unit is used to perform safe classification by analyzing the activation state inside the LLM through training a lightweight classifier, or to identify unsafe concepts activated in the LLM. The latent space manipulation unit is used to inject a controlled vector into the hidden state when an abnormal signal is detected. The controlled vector is strategically calculated to guide the model to generate a safe output.

[0015] Beneficial effects: By intervening in the generation process in real time rather than intercepting it afterward, the generation path of harmful content is blocked at the source, thus achieving security control while ensuring text quality and semantic coherence.

[0016] Furthermore, the offline optimization module includes: The recording unit is used to store the dialogue history and the detection results of each protection module; The evaluation unit is used to evaluate the detection accuracy of the input / output module, as well as the security and generation quality of the response, using automated or semi-automated methods. It can be aided by retrieval-enhanced generation technology for analysis. The dataset building unit is used to build new datasets from insecure user requests that successfully bypass the protection system, secure user requests that are mistakenly intercepted by the protection module, and responses that are unrelated to the user's intent, and to expand the datasets by combining data augmentation techniques. The optimization unit is used to enhance defense capabilities by adjusting model parameters, improving module architecture, or updating system memory using new datasets, and to prevent forgetting by employing a phased continuous learning approach.

[0017] Beneficial effects: By automatically collecting quality feedback on adversarial examples and generated content during online defense through a closed-loop feedback mechanism, the defense capability can be continuously enhanced without relying on manual annotation and full model retraining.

[0018] Option 2: A content security filtering method for large language models, comprising the following steps: Online reasoning stage: Upon receiving a user request, the input protection module performs feature extraction based on a small language model, and uses an adaptive linear detection mechanism on the optimal hidden state to perform binary classification, labeling the prompt words as safe or unsafe. Unsafe warning words are sent to the multi-label classification module for violation type marking. Based on the marking results, a corresponding response template is provided or the transmission to the LLM model is blocked. During the LLM model's response generation process, an internal abnormal signal is monitored in real time through a controllable generation module, and the model is guided to generate a safe output by adjusting the latent space vector or the probability distribution of generated words. The output protection module performs final detection and purification on the generated response, intercepts unsafe responses, and replaces them with a preset safe response template to return to the user. Offline optimization phase: Record the dialogue history and the test results of each protection module; The performance of the protection module is evaluated using automated or semi-automated methods, and the analysis is aided by search-enhanced generation technology. New datasets are built from successfully bypassed attacks, mistakenly intercepted normal requests, and irrelevant responses, and expanded using data augmentation techniques. The defense module is optimized using a new dataset, and a hybrid learning approach using both new and old samples is adopted to prevent forgetting. This ensures stable performance on previous tasks when new knowledge is injected.

[0019] Beneficial effects: Through the two-stage collaboration of online inference and offline optimization, the defense capabilities can be continuously evolved while ensuring low latency in real-time services, enabling the system to adapt to new attacks and harmful content without interrupting services.

[0020] Furthermore, the method for determining the optimal hidden state includes: Feature representations for each layer are obtained based on a pre-trained small language model; Train a classifier for each layer to perform binary classification of harmful content, and test the classifier on a validation set; Select the hidden layer with the highest F1 score as the optimal hidden layer; During expert evolution, when fine-tuning the pre-trained model that acquires feature representations, the optimal location of the hidden layer is re-found.

[0021] Beneficial effects: By using data-driven optimal hidden layer selection, the semantic understanding capabilities of the language model are fully utilized to capture more relevant semantic information for specific tasks, saving inference time and improving detection accuracy.

[0022] Furthermore, the quantification conditions and priorities for expert evolution include: Evolution is triggered when the attack success rate and false alarm rate are higher than preset thresholds. The attack success rate represents the proportion of malicious queries that successfully bypass the defense system, and the false alarm rate represents the proportion of normal queries that are misclassified as malicious instructions. The evolution priority is as follows: mix the new dataset with the old data and fine-tune the expert layers with low training accuracy; if this does not meet the requirements, fine-tune the pre-trained model that obtains feature representations and each expert layer and find the best hidden layer again; if this still does not meet the requirements, add expert models and adjust the router's classification.

[0023] Beneficial effects: By using quantified evolutionary triggering conditions and incremental optimization strategies, an optimal balance is achieved between computational resource consumption and improved defense performance, avoiding unnecessary expansion of system complexity.

[0024] Furthermore, the intervention method of the controllable generation module includes: The abnormal signal is obtained by real-time security classification of the internal activation state of the LLM using a lightweight classifier. The lightweight classifier and the input protection module share the same feature extraction layer of the pre-trained small language model. The injection location of the controlled vector is determined based on the hidden layer number where the abnormal signal is first detected. Injection is performed only in the layer with that number and the adjacent next layer, without interfering with other hidden layers.

[0025] Beneficial effects: By sharing feature extraction layers, the number of model loads is reduced, thus lowering memory usage; by limiting the intervention scope to the layer containing the anomalous signal and its adjacent layers, computational redundancy and generation quality degradation caused by full-layer intervention are avoided, improving intervention accuracy while maintaining low latency. By implementing intervention during the decoding stage rather than modifying model weights, fine-grained safety control is achieved without affecting the general capabilities of LLM, avoiding model performance degradation caused by traditional safety fine-tuning.

[0026] The advantages of this invention are: This invention breaks through the industry's widely accepted trilemma of impossibility bias in terms of accuracy, efficiency, and evolution by reconstructing two underlying mechanisms: optimal hidden state adaptive linear probing and evolutionary multi-expert architecture.

[0027] An unexpected solution to the technical problem: Traditionally, strong detection capabilities are inevitably accompanied by high latency and high computational costs, while continuous adaptation to new threats inevitably relies on frequent full-model retraining. This invention unexpectedly discovers that by performing adaptive linear probing on the optimal hidden layer of a small language model, detection accuracy surpassing that of general-purpose large models can be achieved while maintaining extremely low inference latency. Simultaneously, through a closed-loop feedback-driven progressive expert evolution, the system can continuously adapt to emerging attack patterns without interrupting online service or expanding the model size, completely changing the established technical paradigm that evolution inevitably involves high costs.

[0028] Unexpected Technological Achievements: This invention transforms the hybrid expert approach from the internal mechanism of the protected object (large language model) into the core architectural paradigm of the protection system itself. This paradigm shift brings unexpected technical effects. Traditional MoE is applied within LLM to improve model capacity and efficiency, while this invention, when applied to the protection system, enables each expert to focus on specific attack types through specialized division of labor. This not only reduces the learning difficulty for individual experts but also allows routing decisions and expert tuning to be optimized independently. At the same time, optimal hidden layer detection provides high-purity feature input to lightweight experts, and the two form a positive reinforcement loop. This combined effect far exceeds the linear superposition of single technologies, achieving a synergistic gain of 1+1>2. Attached Figure Description

[0029] Figure 1 This is a general framework diagram of the system of the present invention.

[0030] Figure 2 This is a schematic diagram illustrating the use of the input protection module in the system of this invention.

[0031] Figure 3 This is a schematic diagram illustrating the use of the controllable generation module in the system of the present invention. Detailed Implementation

[0032] The following detailed description illustrates the specific implementation method: The large language model content security filtering system of this invention ensures the security of the large language model through a two-stage process of online inference and offline optimization. The online inference stage implements a multi-layered protection mechanism for user requests to ensure secure responses; the offline optimization stage continuously optimizes the model based on interaction history, enabling the protection system to adapt to new attacks and harmful content.

[0033] Figure 1 This diagram illustrates the overall framework of the system of the present invention, showcasing its multi-layered protection structure. The architecture comprises two phases: an online inference phase and an offline optimization phase.

[0034] The online inference phase includes three protection modules on the LLM server side: (1) Input protection module: This module achieves low latency and strong detection capability (low false alarm rate and false negative rate), while detecting and blocking unsafe prompts containing harmful, unethical or policy-violating content, preventing them from being transmitted to the LLM model or returned to the database's preset security response.

[0035] (2) Controllable Generation Module: This module monitors abnormal signals within the LLM model in real time during the response generation process (such as activation states in the hidden layers or intermediate outputs like the attention matrix), and guides the model to generate safe outputs by adjusting the latent space vectors or the probability distribution of generated words. Controllable generation must simultaneously ensure efficiency, text quality, and security.

[0036] (3) Output protection module: As the final barrier of the protection system, this module detects and eliminates unsafe responses with low latency and high detection accuracy (low false alarm rate and false negative rate).

[0037] The offline optimization phase on the LLM server side includes four steps: (1) Record dialogue and defense results: The server stores the dialogue history and the detection results of each protection module.

[0038] (2) Evaluation of protection module performance: The evaluation is conducted using automated / semi-automated methods, with the aid of search-enhanced generation technology. The focus is on evaluating the detection accuracy of the input / output modules, as well as the security and generation quality of the response.

[0039] (3) Constructing a new dataset: Data sources include: insecure user requests that successfully bypass the protection system; secure user requests that are mistakenly intercepted by the protection module; and responses that are unrelated to the user's intent. Data augmentation techniques can be used to construct the dataset.

[0040] (4) Optimize the defense module: Improve defense capabilities by adjusting model parameters, improving module architecture, or updating system memory, using new datasets. When injecting new knowledge, it is necessary to ensure that the performance of the protection system remains stable for previous tasks.

[0041] Figure 2 This paper demonstrates an example of the use of the input protection module in the system of this invention, which simultaneously achieves low latency and strong detection capabilities. The input protection module comprises two components: binary classification and multi-label classification. The binary classification labels prompts as safe / unsafe, and unsafe prompts are sent to the multi-label classification module for violation type marking. Each component employs an expert hybrid structure to route user prompts to the most suitable classifier.

[0042] Figure 3 This paper demonstrates an example of the use of the controllable generation module in the system of this invention, which generates safe responses through latent space manipulation. During text generation, the server monitors anomalous signals in the internal state of the LLM (e.g., analyzing activation states for safe classification by training a lightweight classifier, or identifying unsafe concepts activated in the LLM). Once an anomalous signal is detected, the server injects controlled vectors into the hidden state; these strategically computed vectors guide the model to generate safe outputs.

[0043] The main prior art involved in the system of this invention is described below: Controlled text generation: Using existing technologies, this invention guides the output of a language model to conform to preset attributes (such as style, theme, and security). This invention focuses solely on control techniques during the inference phase, including latent space manipulation to adjust the activation state of the model's hidden layers, and decoding-time regulation to intervene during the decoding process.

[0044] Retrieval-enhanced generation: This technique enhances LLM output by retrieving information from external knowledge sources and integrating it. Key components include index building, information retrieval, content enhancement, and response generation. This technology supports real-time knowledge updates and is particularly suitable for dynamically evolving systems.

[0045] Small language models: Small language models, typically with hundreds of millions to billions of parameters, offer computational advantages in deployment scenarios requiring low latency and low-cost inference. Recent research shows that specialized small models optimized with properly trained data can outperform general-purpose large models on specific tasks, demonstrating significant potential in the field of security auditing.

[0046] Linear Probing in Hidden Layers: This technique fully leverages the semantic understanding capabilities of language models by training a penalized classifier on frozen model embeddings. The method trains the classifier on features from the optimal hidden layer outputs, layers that capture more task-relevant semantic information and save inference time. Layer-enhanced classification methods have demonstrated that linear probing on small language models achieves state-of-the-art performance in security detection tasks, outperforming general-purpose large models with only a small number of training samples. However, these static methods operate based on the assumption of a fixed threat distribution, failing to address key challenges in security system lifecycle management and lacking the ability to adapt to dynamic adversarial environments without complete retraining.

[0047] Hybrid Expert Architecture: This system employs multiple specialized models, with each expert focusing on a specific input type, and expert selection is determined through a routing mechanism. While activating only relevant experts can improve efficiency, traditional hybrid expert systems face challenges in determining the optimal number of experts: too few experts lead to insufficient specialization when dealing with diverse threats, while too many experts introduce unnecessary computational overhead and system complexity with limited performance improvements.

[0048] The overall R&D concept of this invention aims to overcome the long-standing "accuracy-efficiency-evolution" impossibility triangle bias in the field of large-model security protection. This bias stems from the industry's general belief that strong detection capabilities inevitably come with high latency and high computational overhead, while continuous adaptation to new threats inevitably relies on frequent full-model retraining. This invention achieves a breakthrough through a disruptive reconstruction of two underlying mechanisms: First, it proposes "optimal hidden state adaptive linear detection," breaking the inertia of fixed-level feature extraction. During inference, it dynamically selects the intermediate layer with the highest security information concentration within the large language model for detection, obtaining deep semantic accuracy at the cost of shallow computation, thus changing the technical bias that "accuracy is positively correlated with computational power." Second, it proposes an "evolutionary multi-expert architecture," transforming the hybrid expert approach from the internal mechanism of the protected object into the core architectural paradigm of the protection system itself. Through a mechanism driven by specialized division of labor and closed-loop feedback, the system can continuously adapt to emerging jailbreak attack patterns without expanding the model size or interrupting online services. "Optimal hidden layer detection" provides high-purity feature input to lightweight experts, reducing the learning difficulty for experts.

[0049] The specific technical details adopted in this invention include: 1. Specific specifications of a small language model Based on the pre-trained Qwen2.5-0.5B model, the feature representation of the optimal hidden layer is obtained for classification training. However, the method of this invention is not limited to this specific model and can be extended to other models.

[0050] 2. Mathematical definition of the optimal hidden state Obtain the feature representation of each layer (B is the batch size, T is the sequence length, and d is the feature dimension), and mean pooling is performed on the sequence dimension to obtain... For each layer, a classifier is trained to perform binary classification of whether the content is harmful or not; then the classifier is tested on the validation set, and the hidden layer with the highest F1 score is selected as the optimal hidden layer.

[0051] 3. The specific structure of the expert classifier First, feature representations (vectors) are obtained from the pre-trained model, and then classification is performed based on a linear classifier. Currently, each expert uses a different linear classifier, but this can be extended to a shallow MLP. This way, when loading multiple experts simultaneously, high memory usage is not required.

[0052] 4. Implementation of the routing module A small language model (Qwen2.5-0.5B) is trained for classification. This language model outputs the probability for each expert model. If the confidence score (the highest probability value) is greater than a preset threshold, the prompt word is sent to the expert model corresponding to the highest probability value; otherwise, all expert models are integrated for classification, i.e., their results are averaged through voting. The threshold is set according to specific requirements.

[0053] 5. Quantitative conditions for expert evolution When the defense accuracy of experts on a new subset of data exceeds a certain threshold, namely, when the attack success rate and the false positive rate (the proportion of normal queries being misclassified as harmful instructions) are higher than a certain threshold (currently the thresholds for attack success rate and false positive rate are 10% and 8% respectively, but this can be adjusted according to specific circumstances), the evolutionary conditions will be triggered.

[0054] The evolutionary priorities are as follows: First, fine-tune each expert layer only: mix the new dataset with the old data used, and fine-tune the expert layers (linear classifiers or shallow MLPs) that have low training accuracy.

[0055] Then, if fine-tuning the expert layers does not meet the requirements, the pre-trained model that obtains feature representations will be fine-tuned along with each expert layer. During this process, the optimal position of the hidden layers will be re-searched.

[0056] If the above two steps still do not meet the requirements, the router and expert model classifications will be adjusted, such as adding an expert model and adjusting the router classifications.

[0057] 6. Direct experimental data Dataset: The test dataset for this invention was constructed by sampling from two publicly available datasets. Harmful queries were sampled from WildGuardMix, and normal queries were sampled from LMSYS-Chat-1M, resulting in 2,500 normal queries and 3,000 harmful queries.

[0058] Model: Three expert models were used in the test. Both the expert models and the routing model were based on the Qwen2.5-0.5B model.

[0059] Benchmark: As shown in Table 1, this invention is compared with Llama Guard38B, mainly by comparing the F1 score and latency of the input classifier.

[0060] Table 1

[0061] 7. Mechanisms to prevent forgetting in continuous learning To achieve the goal of continuous learning and preventing forgetting, the offline optimization module maintains a historical sample storage unit. This unit stores representative samples generated during online operation. Each sample records the following information (including but not limited to): sample text, manually or automatically confirmed security label, violation category, sample source, triggered protection module, assigned expert ID, classification result of the model output before update, output confidence score, and expert assignment probability of the routing module. The system stores historical samples hierarchically based on the above information. For example, high-risk bypass samples, false positive samples, boundary samples, and typical correct samples are stored separately and extracted according to a preset ratio during subsequent training. Thus, when learning new samples, the system can simultaneously review key historical samples, reducing forgetting caused by continuous learning. Different update mechanisms are adopted according to three different scenarios: (1) When only fine-tuning the expert layer with low training accuracy, the system adopts a forgetting prevention method of updating the target expert individually. Specifically, the system first determines the target expert that needs to be updated based on the violation category of the new sample, the routing result, and the expert confidence, while keeping the parameters of other experts, the routing module, and the feature extraction model unchanged. Subsequently, the system extracts old samples related to the target expert from the historical sample storage unit. The so-called related to the target expert means that these old samples were routed to the expert in the historical inference process, or their violation category belongs to the category that the expert is responsible for, or the expert has made low confidence or incorrect judgments on these samples. The system uses these old samples and new samples together to form training data to continuously learn the target expert.

[0062] (2) When fine-tuning only the expert layer is insufficient, and it is necessary to fine-tune the pre-trained small language model used to obtain feature representations and each expert layer, and to re-find the optimal hidden layer, the system further employs a feature representation preservation mechanism. This means that the hidden layer feature representations of the fine-tuned model on historical samples must remain close to those before fine-tuning. Before training, the system saves the hidden layer feature representations, classification outputs, and optimal hidden layer positions of the old model on historical samples. During joint fine-tuning, the system not only uses new and old samples for classification training, but also requires that the hidden layer feature representations of the fine-tuned model on historical samples maintain a close distance or cosine similarity to the feature vectors before fine-tuning.

[0063] When searching for the optimal hidden layer, the system considers both the performance on new samples and the performance on old samples. Specifically, the system trains or evaluates a classifier for each candidate hidden layer and calculates its performance on new attack samples, historical attack samples, historical normal requests, and boundary samples. Only when a candidate hidden layer improves the recognition ability of new samples without significantly decreasing the recall rate of historical attacks or significantly increasing the false positive rate of normal requests, and its overall performance meets a preset threshold, is it determined as the new optimal hidden layer.

[0064] (3) When fine-tuning the feature extraction model and expert layer still fails to meet the requirements, and it is necessary to add expert models and adjust router classification, the system adopts an expert retention and route stabilization mechanism. Specifically, the system does not delete the original experts, but retains the parameters of the original experts and trains new experts. The newly added experts are mainly used to learn new attack samples, boundary samples, or newly emerging false alarm samples that are difficult for the original experts to identify. When training new experts, the system mainly uses new attack samples and their enhanced samples, while introducing some historical boundary samples to determine the division of labor between new experts and original experts. When adjusting the router, the system records the expert number and route probability assigned to the historical samples before the update. For historical samples, penalties such as KL divergence are added to ensure that the updated router should maintain the original allocation results as much as possible.

[0065] 8. Closed-loop feedback evaluation system The evaluation is primarily based on attack success rate and false positive rate: Attack success rate represents the percentage of malicious queries that successfully bypass defense systems.

[0066] The false alarm rate represents the proportion of legitimate queries that are misclassified as malicious instructions.

[0067] The specific thresholds for attack success rate and false positive rate can be adjusted appropriately according to the specific circumstances. Their specific details are not related to the technical problem solved by this invention and will not be elaborated here.

[0068] 9. Collaborative Mechanism of Pre / Post Classifiers The present invention currently trains separately according to existing technology, and its specific content is not related to the technical problem solved by the present invention, so it will not be described in detail here.

[0069] 10. Integration interface with the main LLM Using existing technology, after the pre-classifier intercepts the prompt word, it is fed into a multi-classifier to classify it into harmful categories; based on the harmful category of the prompt word, a corresponding response template is provided.

[0070] After the classifier intercepts the response, it discards it and replaces it with a preset safe response template to return to the user.

[0071] The differences and advantages of this invention compared with prior art document 1 (CN119808883A, Risk Defense Method and Device for Large Language Model Jailbreak Attacks): Comparison document 1 constructs a defense-in-depth system from the input side, model side, and output side. Its input side adopts a dynamic malicious keyword library and intent recognition. The model side dynamically adjusts by adding network layers / neurons and uses GAN to generate adversarial examples for adversarial training. The output side constructs an output review library for content purification.

[0072] This invention differs fundamentally from prior art document 1 in the following ways: First, prior art document 1's input-side defense relies on shallow features such as keyword matching and sentiment analysis, while this invention employs optimal hidden state adaptive linear probing to extract security features from the deep semantic representation of the LLM. The detection dimension extends from surface text to the semantic latent space, providing stronger identification capabilities against covert attacks that have undergone semantic transformation. Second, prior art document 1's model-side dynamic adjustments directly modify the network structure and parameters of the protected LLM, posing an inherent risk of compromising the model's general capabilities. Furthermore, each adjustment requires service interruption and retraining. In contrast, this invention's controllable generation module only intervenes in the latent space vector during the inference phase to guide the output without modifying the LLM weights, ensuring that the model's general capabilities remain unaffected and without service interruption. Third, prior art document 1 lacks a systematic and continuous evolution mechanism; its dynamic keyword library expansion and adversarial training both rely on manual triggering and batch processing. In contrast, this invention achieves automated and continuous self-enhancement of defense capabilities through a closed-loop feedback mechanism of recording, evaluation, optimization, and deployment, completing a paradigm shift from passive response to proactive evolution.

[0073] The differences and advantages of this invention compared to prior art document 2 (CN202410592521.4, Large Model Security Protection Method, Computer Program Product and Server): Comparison document 2 focuses on post-attack interception and tracing. It uses the RASP module to hook access to basic functions of the large model and implements request-response closed-loop verification based on the consistency check of user identifier / session identifier. The core is to achieve access control and attack tracing through certificate encryption and m-sequence scrambling.

[0074] The technical approach of this invention differs significantly from that of prior art document 2: Prior art document 2 is an access control and auditing traceability scheme at the security architecture level, whose technical essence is to prevent unauthorized access through identity authentication and request-response matching, without involving security semantic analysis of the prompt content itself; while this invention is an intelligent filtering scheme at the content security level, directly performing deep security detection on the semantic content of the prompts and responses. Specifically, prior art document 2 cannot identify jailbreak attacks initiated by legitimate users (because their user identities are legitimate), while the input protection module of this invention can accurately identify attack semantics; the source tracing capability of prior art document 2 relies on post-event analysis, while the offline optimization module of this invention achieves pre-emptive enhancement of defense capabilities by collecting adversarial samples in real time; the RASP mechanism of prior art document 2 introduces additional communication and computational overhead, while this invention achieves low-latency deployment based on a lightweight architecture with a small language model.

[0075] The differences and advantages of this invention compared to prior art document 3 (CN202510104813.3, Static Application Security Detection False Alarm Judgment System Based on LLM): Comparison document 3 addresses false positive detection in static application security detection. It trains a large language interpretation model to interpret security detection rules in natural language through BERT vectorization, similarity calculation, and vector library construction. The core of this approach is to reduce the understanding threshold of security rules and improve the efficiency of false positive detection.

[0076] The application scenarios and technical objectives of this invention differ fundamentally from those of prior art document 3: Prior art document 3 addresses false positive detection for static security rules, and its technical solution revolves around the vectorization and similarity matching of rule texts, belonging to a security operation efficiency tool; while this invention addresses real-time content security filtering in dynamic interaction processes, belonging to a core security component of large language model online services. Technically, prior art document 3 uses a static BERT model for vectorization, lacking adaptability to new attacks, and its large language interpretation model requires manual annotation and training, resulting in high update costs; while this invention employs an evolutionary multi-expert architecture and a closed-loop feedback mechanism, enabling it to automatically adapt to attack evolution, and its optimal hidden layer detection based on a small language model is far more computationally efficient than the BERT full-model inference in prior art document 3.

[0077] The differences and advantages of this invention compared to conventional techniques in the field: Conventional security measures in this field include rule-based keyword filtering, traditional machine learning-based classifier detection, and content moderation based on large language models. This invention overcomes the inherent limitations of these three approaches: Compared to rule-based systems, this invention addresses the issue of insufficient adaptability to new attacks through deep semantic detection and continuous evolution capabilities; compared to traditional machine learning classifiers, this invention fully utilizes the semantic understanding capabilities of pre-trained language models through optimal hidden layer detection, achieving leading performance with a small number of samples and avoiding the need for large amounts of labeled data and frequent retraining; compared to LLM-based moderation schemes, this invention reduces inference latency by an order of magnitude through a hybrid architecture of small language models and experts, while achieving a technological leap from post-event interception to in-event guidance through a controllable generation module, significantly improving user experience and system throughput while ensuring security.

[0078] Specifically, Example 1: Deployment of an input protection module based on Qwen2.5-0.5B In this embodiment, the input protection module is built based on a pre-trained Qwen2.5-0.5B model. The specific implementation steps are as follows: Obtain the feature representation of each layer of the Qwen2.5-0.5B model. Where B=32, T is the length of the input sequence, and d=896. Mean pooling over the sequence is performed on the feature representations of each layer to obtain... The linear classifier was trained layer by layer on the validation set and evaluated using F1 score. The 18th layer was determined to be the optimal hidden layer (F1=0.823), which was significantly higher than the first layer (F1=0.612) and the last layer (F1=0.754).

[0079] Based on the optimal hidden layer features, three expert classifiers are deployed: a general harmful content expert, a jailbreak attack expert, and a social engineering attack expert. The routing module is also built based on Qwen2.5-0.5B, outputting the probability distribution of the three experts, with a confidence threshold set to 0.75. When the confidence of a single expert exceeds the threshold, the route is directly routed to the corresponding expert; otherwise, a voting average is performed.

[0080] Dataset Construction: 3000 harmful queries were sampled from WildGuardMix, and 2500 normal queries were sampled from LMSYS-Chat-1M. Compared with the benchmark Llama Guard38B (F1=0.772, latency 289.98ms), the input protection module in this embodiment outperforms Llama Guard38B in F1 score and has a latency far below 289.98ms, achieving an order-of-magnitude efficiency improvement while maintaining stronger detection capabilities.

[0081] Example Results: This example verifies the feasibility of optimal hidden layer detection based on a small language model, proving that through clever feature extraction layer selection, a lightweight model can outperform general large models on specific security detection tasks, breaking the industry consensus that model size determines detection capability.

[0082] Example 2: Closed-Loop Feedback Driven Expert Evolution In this embodiment, after the system had been running for 30 days, the attack success rate was found to have increased to 12% and the false alarm rate to 9%, triggering the evolution condition.

[0083] Phase 1 Evolution: 150 successfully bypassed attack samples were mixed with 3000 randomly sampled old samples from historical data. Only the linear classifier layer of the jailbreak attack expert was fine-tuned. After fine-tuning, the expert's recall rate on attack samples increased from 68% to 89%, the overall attack success rate decreased to 7%, and the false positive rate decreased to 5%, meeting the performance requirements. The evolution then terminated.

[0084] If the first stage fails to meet the requirements (the attack success rate is still higher than 10%), then the second stage begins: the pre-trained model parameters are unfrozen, fine-tuned jointly with the expert layer, and the optimal hidden layer is searched again. At this point, the optimal hidden layer is found to have migrated to layer 22, and with joint fine-tuning, the attack success rate is reduced to 6%.

[0085] If the second phase still fails to meet the requirements, the third phase begins: a fourth expert is added (to counter multi-turn dialogue attacks), and the router is retrained to support 4-way classification. The new expert is initialized based on the data accumulated in the first two phases, quickly reaching a usable state.

[0086] Example Results: This example demonstrates the optimal balance between computational resource consumption and defensive performance in a progressive evolutionary strategy. Compared to full model retraining, which requires hours to days and service interruption, the first stage of evolution in this example takes only minutes and does not affect online service. New experts added in the third stage can also be seamlessly integrated via hot-loading, achieving the engineering goal of uninterrupted service during evolution.

[0087] Example 3: Latent Space Intervention of Controllable Generation Module In this embodiment, when the input protection module approves an edge safety request (such as "Please describe the chemical experiment steps in detail, including possible safety risks"), the LLM gradually shifts towards outputting the synthesis details of hazardous chemicals during the generation process.

[0088] The lightweight classifier in the controllable generation module monitors the hidden state of LLM layer 15 in real time and detects an abnormally enhanced activation pattern of neurons related to the concept of "synthesis of hazardous chemicals". At this point, a controlled vector v = -0.3 × g is injected into this hidden state, where g is the direction vector of the concept of "synthesis of hazardous chemicals" (extracted in advance using concept activation vector technology). This intervention causes the model to reduce the generation probability of related terms in subsequent decoding and instead output safety guidance content such as "This experiment must be performed by certified personnel in a professional laboratory; the following is only an overview of the principle".

[0089] The output after intervention is verified a second time by the output protection module. Once confirmed as a safe response, it is returned to the user. The entire intervention process adds a delay of << 5ms, which is imperceptible to the user.

[0090] Example Results: This example demonstrates the superiority of in-process intervention over post-process interception. Traditional solutions require full-text review and rewriting after generation, which significantly increases response delay and may disrupt semantic coherence. In contrast, the latent space intervention in this example guides the generation direction from the source, maintaining text quality and user experience while ensuring security, achieving a triple unity of efficiency, quality, and security.

[0091] Example 4: Anti-forgetting mechanism for offline optimization module In this embodiment, after 90 days of system operation, a total of 420 successfully bypassed samples, 180 falsely intercepted samples, and 80 irrelevant responses were collected. When constructing a new dataset, data augmentation techniques were used to expand the bypass samples by paraphrasing, sentence transformation, and inserting noise, generating 1680 augmented samples.

[0092] During optimization, the new dataset was mixed with 5000 old samples randomly sampled from historical data at a 1:3 ratio for training. To verify the anti-forgetting effect, the system's F1 score on the initial test set was tested before and after optimization: 0.831 before optimization and 0.828 after optimization, with a performance degradation of << 0.5%, meeting the stability requirements. If trained with entirely new data, the F1 score plummeted to 0.743, indicating significant forgetting.

[0093] Example Results: This example demonstrates the crucial role of hybrid learning strategies in continuous learning scenarios. In the dynamically evolving field of large language model security protection, new attack patterns are constantly emerging. Focusing solely on new samples will lead to a degradation in defense capabilities against older attack types. The hybrid learning mechanism of this invention ensures the cumulative growth rather than replacement of defensive knowledge, enabling the system to possess true "lifelong learning" capabilities.

[0094] The above descriptions are merely embodiments of the present invention, and common knowledge such as specific technical solutions and / or characteristics are not described in detail here. It should be noted that those skilled in the art can make various modifications and improvements without departing from the technical solutions of the present invention, and these should also be considered within the scope of protection of the present invention. These modifications and improvements will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. A content security filtering system for large language models, characterized in that, include: An input protection module, configured on the LLM server, receives user requests, extracts features from the user requests based on a small language model, performs binary classification using an adaptive linear probing mechanism on the optimal hidden state, labels prompt words as safe or unsafe, and sends unsafe prompt words to a multi-label classification module for violation type marking. Internally, the input protection module employs an evolutionary multi-expert architecture, using a routing module to assign user prompt words to the most suitable expert classifier. Each expert classifier focuses on a specific input type and continuously evolves to detect emerging harmful prompts and new attack patterns. The controllable generation module, configured on the LLM server, is used to monitor abnormal signals inside the LLM model in real time during the response generation process. By adjusting the probability distribution of the latent space vector or generated lexical units, it guides the model to generate safe output. An output protection module, configured on the LLM server, serves as the final barrier of the protection system. It detects and eliminates unsafe responses with low latency and high detection accuracy. Furthermore, the output protection module can continuously evolve to detect emerging harmful responses. The offline optimization module, configured on the LLM server, is used to continuously record the dialogue history and the detection results of each protection module. It uses automated or semi-automated methods to evaluate the performance of the protection modules, builds a model optimization dataset, and optimizes the defense modules by adjusting model parameters, improving module architecture, or updating system memory. When injecting new knowledge, it ensures that the performance of the protection system remains stable for previous tasks.

2. The large language model content security filtering system according to claim 1, characterized in that, The input protection module includes: A binary classification component is used to label prompts as safe or unsafe. A multi-tag classification component is used to mark unsafe warning words as violations. Both the binary classification component and the multi-label classification component employ an expert hybrid structure to route user prompts to the most suitable classifier.

3. The large language model content security filtering system according to claim 1, characterized in that, The adaptive linear detection mechanism on the optimal hidden state includes: Obtain feature representations of each layer of a pre-trained small language model Where B is the batch size, T is the sequence length, and d is the feature dimension; The feature representation is subjected to mean pooling along the sequence dimension to obtain the pooled layer feature vector. ; Train a classifier for each layer to perform binary classification of harmful content, and test the classifier on the validation set. Select the hidden layer with the highest F1 score as the optimal hidden layer. The position of the optimal hidden layer is dynamically adjusted during the expert evolution process.

4. The large language model content security filtering system according to claim 1, characterized in that, The evolved multi-expert architecture includes: Multiple expert classifiers, each expert being a different linear classifier or a shallow MLP; The routing module classifies based on a small language model and outputs the probability for each expert model. If the confidence level is greater than a preset threshold, the prompt word is sent to the expert model with the highest probability value; if it does not exceed the threshold, all expert models are integrated for classification, and their results are averaged by voting. The expert classifier triggers evolution when the defense accuracy exceeds the threshold. The evolution priority is as follows: fine-tuning only each expert layer, fine-tuning the pre-trained model that obtains feature representations and each expert layer and re-finding the best hidden layer, and adjusting the router and expert model classification.

5. The large language model content security filtering system according to claim 1, characterized in that, The controllable generation module includes: An abnormal signal monitoring unit is used to perform safe classification by analyzing the activation state inside the LLM through training a lightweight classifier, or to identify unsafe concepts activated in the LLM. The latent space manipulation unit is used to inject a controlled vector into the hidden state when an abnormal signal is detected. The controlled vector is strategically calculated to guide the model to generate a safe output.

6. The large language model content security filtering system according to claim 1, characterized in that, The offline optimization module includes: The recording unit is used to store the dialogue history and the detection results of each protection module; The evaluation unit is used to evaluate the detection accuracy of the input / output module, as well as the security and generation quality of the response, using automated or semi-automated methods. It can be aided by retrieval-enhanced generation technology for analysis. The dataset building unit is used to build new datasets from insecure user requests that successfully bypass the protection system, secure user requests that are mistakenly intercepted by the protection module, and responses that are unrelated to the user's intent, and to expand the datasets by combining data augmentation techniques. The optimization unit is used to enhance defense capabilities by adjusting model parameters, improving module architecture, or updating system memory using new datasets, and to prevent forgetting by employing a phased continuous learning approach.

7. A method for content security filtering of large language models, characterized in that, Includes the following steps: Online reasoning stage: Upon receiving a user request, the input protection module performs feature extraction based on a small language model, and uses an adaptive linear detection mechanism on the optimal hidden state to perform binary classification, labeling the prompt words as safe or unsafe. Unsafe warning words are sent to the multi-label classification module for violation type marking. Based on the marking results, a corresponding response template is provided or the transmission to the LLM model is blocked. During the LLM model's response generation process, an internal abnormal signal is monitored in real time through a controllable generation module, and the model is guided to generate a safe output by adjusting the latent space vector or the probability distribution of generated words. The output protection module performs final detection and purification on the generated response, intercepts unsafe responses, and replaces them with a preset safe response template to return to the user. Offline optimization phase: Record the dialogue history and the test results of each protection module; The performance of the protection module is evaluated using automated or semi-automated methods, and the analysis is aided by search-enhanced generation technology. New datasets are built from successfully bypassed attacks, mistakenly intercepted normal requests, and irrelevant responses, and expanded using data augmentation techniques. The defense module is optimized using a new dataset, and a hybrid learning approach using both new and old samples is adopted to prevent forgetting. This ensures stable performance on previous tasks when new knowledge is injected.

8. The content security filtering method for large language models according to claim 7, characterized in that, The method for determining the optimal hidden state includes: Feature representations for each layer are obtained based on a pre-trained small language model; Train a classifier for each layer to perform binary classification of harmful content, and test the classifier on a validation set; Select the hidden layer with the highest F1 score as the optimal hidden layer; During expert evolution, when fine-tuning the pre-trained model that acquires feature representations, the optimal location of the hidden layer is re-found.

9. The content security filtering method for large language models according to claim 7, characterized in that, The quantitative conditions and priorities for expert evolution include: Evolution is triggered when the attack success rate and false alarm rate are higher than preset thresholds. The attack success rate represents the proportion of malicious queries that successfully bypass the defense system, and the false alarm rate represents the proportion of normal queries that are misclassified as malicious instructions. The evolution priority is as follows: mix the new dataset with the old data and fine-tune the expert layers with low training accuracy; if this does not meet the requirements, fine-tune the pre-trained model that obtains feature representations and each expert layer and find the best hidden layer again; if this still does not meet the requirements, add expert models and adjust the router's classification.

10. The content security filtering method for large language models according to claim 7, characterized in that, The intervention methods of the controllable generation module include: The abnormal signal is obtained by real-time security classification of the internal activation state of the LLM using a lightweight classifier. The lightweight classifier and the input protection module share the same feature extraction layer of the pre-trained small language model. The injection location of the controlled vector is determined based on the hidden layer number where the abnormal signal is first detected. Injection is performed only in the layer containing that number and the adjacent next layer, without interfering with other hidden layers.

Images