External large language model security firewall system and control method thereof
By covering both pipelined and end-to-end implementations, and combining gateway and bypass mirroring deployments, the security issues of large language models in streaming output scenarios are solved, security and compliance are improved, the risk of adversarial probing is reduced, and dynamic review and control of candidate outputs are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- 顾聪聪
- Filing Date
- 2026-01-20
- Publication Date
- 2026-05-01
AI Technical Summary
Large language models are vulnerable to jailbreak prompts, segmented inducements, and black-box adversarial probing attacks in scenarios involving streaming output, tool calls, and multi-turn dialogues. Existing solutions have limitations in adaptability and room for circumvention.
It adopts a pipelined and end-to-end implementation, combined with gateway and bypass mirror deployment, supports streaming token window detection and protocol revocation/overwrite commands, intercepts interactive data through the data acquisition module, generates control signals and executes output control through the security processing module, and records reproduction information to update the policy through the reproduction and traceability data module.
It improves security, compliance, and auditability, reduces the risk of adversarial detection and reverse-engineering defense logic, achieves clear definition and dynamic review of candidate outputs, and enhances adversarial protection capabilities.
Smart Images
Figure CN121966981A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence security and network content security technology, and in particular to a large language model security firewall system and its control method for performing secure processing and controlling the output of user-side and main language model interaction data. Background Technology
[0002] Large language models are vulnerable to jailbreak prompts, segmented inducements, and black-box adversarial probing attacks in scenarios involving streaming output, tool calls, and multi-turn dialogues. Existing solutions mostly rely on fixed review links or alignment within the main model, which have insufficient adaptability and room for circumvention, such as bypass mirroring deployment, inference pipeline hook deployment, and end-to-end secure rewriting. Summary of the Invention
[0003] This invention aims to provide a security control system and method that covers pipelined and end-to-end implementations, gateway and bypass mirror deployments, and supports streaming token window detection and protocol revocation / override instructions, in order to improve security, compliance and auditability and reduce the risk of adversarial detection and reverse-engineering defense logic.
[0004] Technical solution The system of this invention includes at least a data acquisition module, a security processing subsystem, an output control module, and a reproduction and traceability data module. The data acquisition module is used to intercept, receive, or mirror user input and / or candidate output; the security processing subsystem generates control signals and / or transformed content data based on the interactive data; the output control module responds to the control signals by terminating transmission, replacing / masking, outputting transformed content data, or sending protocol instructions containing withdrawal, overwriting, or rendering failure flags; the reproduction and traceability data module records reproduction information for playback reproduction, policy rollback, or scheduling parameter updates. In a preferred embodiment, it also includes an input-side constraint injection module, used to inject security constraint vectors and / or security prompts into the main language model input side to indirectly affect the main language model output.
[0005] Beneficial effects: (1) The output control closed loop is described as "control signal or transformed content data", covering two types of implementations: direct secure rewrite (Seq2Seq) and instruction-based blocking; (2) Candidate outputs are explicitly defined as complete text blocks, token stream fragments, windowed token sequences or intermediate layer vector representations to avoid fragment detection circumvention; (3) Hiding the displayed content is limited to protocol instructions carrying withdrawal / overwrite / rendering failure flags, defining the implementable technical means; (4) Situational data is limited to structured objects, while allowing multiple forms such as Boolean risk bits / scores / tags / rule IDs / vector fields to avoid adversaries circumventing with simplified data structures; (5) Dynamic review paths are expressed in routing methods such as PRNG random seeds / hash / semantic entropy, taking into account both load balancing and adversarial protection technology effects.
[0006] Terminology and Data Structure Definitions. Candidate output refers to the output objects to be reviewed during the generation process of the main language model. Its granularity includes at least one of the following: complete response text blocks; data fragments in streaming output (e.g., token stream fragments or fragments segmented by sentence); token sequences divided by time windows or token windows; or intermediate layer vector representations of candidate output (e.g., logits vectors, hidden state vectors, or embedding vectors). Control signals are machine-readable signals used to drive the output control module, including blocking, rewriting, degradation, or withdrawal / overwrite instruction categories. Transformed content data is the secure output content obtained by the security processing subsystem through filtering, rewriting, replacing, or summarizing the candidate output, and can be directly used as the final response output. Protocol instructions are application-layer instructions carrying withdrawal, overwrite, or rendering failure flags. They can be sent to the client or caller via HTTP response header fields, WebSocket messages, Server-Sent Events messages, or application-layer custom protocols to trigger the client to withdraw displayed fragments, overwrite displayed fragments, or invalidate the rendering of displayed fragments. Situational data is a data object used to transmit risk signals between the security detection unit, the adjudication unit, and the fusion decision algorithm. In a pipelined implementation, the situational data is preferably a structured data object (JSON / key-value pairs / scoring tables / vectors / Protobuf, etc.), which contains at least one of the following: Boolean risk bits, risk scores, risk labels, rule hit identifiers, and multidimensional feature vectors; and may contain at least three items from the fields {module_id, risk_score, risk_label, conflict_metric, threshold_version}.
[0007] Fusion Decision Algorithm (Technical). The fusion decision algorithm is an executable decision processing step used to convert risk signals or situational data into control signals and / or transformed content data according to a preset mapping relationship. It may include: (a) Weight matrix aggregation: performing matrix / vector operations S=W·v on the multidimensional risk score v and comparing it with a threshold set to output a control signal; (b) Rule engine / deterministic automaton: inputting the risk label sequence into a deterministic automaton, triggering a control signal generator when the automaton state jumps to a blocking state or a rewriting state; (c) Conflict deepening: triggering additional detection or additional adjudication when the conflict index exceeds a threshold.
[0008] Key Implementation Points. For streaming output scenarios, the data acquisition module buffers the token stream, and the security processing subsystem can detect the most recent N tokens or the most recent T seconds' output segments based on a sliding window. When the risk score within the window exceeds the threshold or the automaton enters a blocking state, the output control module executes a protocol command to terminate transmission or send a withdrawal / overwrite / rendering failure flag. In another implementation, the security processing subsystem generates transformed content data and outputs the replaced / masked security segments, achieving "generation and cleaning on the go" streaming security output.
[0009] Dynamic Review Path and Random Seed (Preferred Embodiment) In a preferred embodiment, the system sets up a review scheduling module to construct a dynamic review path, dynamically selecting the combination method and calling order of detection units. Random sampling routing is selected based on a random seed generated by a pseudo-random number generator (PRNG). The random seed can be initialized by input risk characteristics, session risk status, timestamps, or policy version identifiers. The dynamic review path is used to achieve load balancing of computing resources and defend against black-box adversarial probing attacks by increasing the uncertainty of the system response. Input-Side Constraint Injection (Preferred Embodiment) In another preferred embodiment, when a preset risk condition is detected, the input-side constraint injection module injects security prompt words or security constraint vectors into the input side of the main language model. For example, security policy constraints are encoded as vectors or template prompt words and concatenated with the original input, thereby enabling the main language model to generate a rejection or rewrite output that conforms to security specifications. This embodiment can be used to cover the technical route of "indirectly controlling output through input-side prompt injection". Attached Figure Description
[0010] Figure 1 This is a schematic diagram of the system architecture (including the security processing subsystem and output control module, covering gateway / bypass mirror / hook-type deployment). Figure 2 This is a schematic diagram of the main flow of the control method; Figure 3 This is a schematic diagram of an embodiment for dynamic review path and random seed selection; Figure 4 A schematic diagram of a multi-turn session risk state machine; Figure 5 This is a typical timing diagram illustrating the interaction between streaming processing and protocol revocation / overwrite / rendering failure commands. Detailed Implementation
[0011] The embodiments of the present invention will be further described below with reference to the accompanying drawings. It should be understood that the following embodiments are for explaining the present invention and not for limiting it. Example 1: End-to-End Secure Rewrite Coverage. The secure processing subsystem employs an end-to-end secure rewriter, with user input and candidate outputs as inputs, and the output being the transformed content data (e.g., securely rewritten text). The output control module directly outputs this secure text, eliminating the need for explicit risk signals or adjudication units, thereby covering the Seq2Seq secure rewrite architecture. Example 2: Input-side prompt injection coverage. After detecting an attack signature fingerprint hit or an escalation of session risk status, the input-side constraint injection module injects security prompts into the input side of the main language model, causing the main language model to generate a rejection or secure rewrite output; at the same time, the reproduction and traceability data module records the injected content and strategy version for reproduction.
Claims
1. A large language model output safety control system, characterized in that, include: — Data acquisition module, used to intercept, receive or mirror data of interaction between the user side and the main language model, wherein the interaction data includes at least user input and / or candidate output of the main language model; — A security processing subsystem for receiving the data from the interaction and generating control signals and / or transformed content data based on at least one security processing strategy, wherein the transformed content data is secure output content obtained by filtering, rewriting, replacing or summarizing candidate outputs; — Output control module, used to output the final response to the client or caller, and to perform at least one of the following operations in response to the control signal and / or the transformed content data: terminate the transmission of the current data stream, replace / mask the output content, output the transformed content data, or send a protocol instruction containing a withdrawal, overwrite or render failure flag to make the client hide or disable the displayed content. — The reproduction and traceability data module is used to record reproduction information for the reproduction and verification process, so as to support playback reproduction, policy rollback or scheduling parameter update; The security processing subsystem can be implemented in any of the following ways: pipelined, end-to-end, or hybrid. — Pipeline implementation includes a set of security detection units and a decision unit; — End-to-end implementations include secure rewriters or risk assessors that perform detection, adjudication, and rewriting within the same model or computational process; — Hybrid implementations include a combination of the two mentioned above.
2. The system according to claim 1, characterized in that, The candidate output includes at least one of the following: a complete response text block, a data fragment of streaming output, a token sequence divided by a time window or a token window, or an intermediate layer vector representation of the candidate output.
3. The system according to claim 1, characterized in that, The security processing subsystem generates a risk signal and maps the risk signal into the control signal through a fusion decision algorithm; the fusion decision algorithm includes at least one of the following: (a) The multidimensional risk scores are weighted and aggregated based on a preset weight matrix and compared with a threshold set to output a control signal; (b) Based on the state transition of the rule engine or deterministic automaton, the risk label sequence is matched and the control signal generator is triggered to output blocking or rewriting control signals.
4. The system according to claim 1, characterized in that, The protocol instructions include at least one or more of the following: withdrawal instructions, overwrite instructions, or rendering failure markers, and are carried and sent via HTTP response header fields, WebSocket messages, Server-SentEvents messages, or application-layer custom protocols.
5. The system according to claim 1, characterized in that, It also includes an input-side constraint injection module, which generates and injects safety constraint vectors and / or safety prompts into the input side of the main language model when a preset risk condition is detected, so that the main language model generates output that conforms to safety specifications; The security constraint vector and / or security prompt words are determined by the risk signal, session risk status, or policy version information output by the security processing subsystem.
6. The system according to claim 1, characterized in that, In the pipelined implementation, the situational data output by the security detection unit set is a structured data object, and its data format is one or more of JSON, key-value pairs, scoring tables, vectors or Protocol Buffers serialization format. The situational data includes at least one of the following: Boolean risk bit, risk score field, risk label field, rule hit identifier field, or multidimensional feature vector field.
7. The system according to claim 6, characterized in that, The situational data includes at least three of the fields module_id, risk_score, risk_label, conflict_metric, and threshold_version, and can be transmitted between processes / containers in serialized form.
8. The system according to claim 1, characterized in that, It also includes a review scheduling module, which is used to determine review configuration parameters and construct dynamic review paths based on input risk characteristics and / or session risk status; wherein the review configuration parameters include at least one or more of the following: number of detection unit calls, parallelism, sampling depth, computation time threshold, threshold set, thread priority parameters, or computing power scheduling strategy.
9. The system according to claim 8, characterized in that, The dynamic review path includes at least one of random sampling routing, hash routing, semantic entropy routing, or rule routing; wherein random sampling routing selects the combination and / or calling order of detection units based on a random seed generated by a pseudo-random number generator, and the random seed is initialized by at least one of input risk features, session risk status, timestamp, or policy version identifier.
10. The system according to claim 9, characterized in that, The dynamic review path is used for at least one of the following technical purposes: to achieve load balancing of computing resources, or to reduce the success rate of jailbreak bypass by increasing the uncertainty of system response to defend against black-box adversarial detection attacks.
11. A control method for the large language model output safety control system as described in claim 1, characterized in that, include: A) Intercept, receive, or mirror user input and / or candidate output; B) The security processing subsystem generates control signals and / or transformed content data based on the data from the interaction; C) The output control module performs at least one operation in response to the control signal and / or the transformed content data: terminating transmission, replacing / masking, outputting the transformed content data, or sending a protocol instruction containing a withdrawal, overwrite, or rendering failure flags. D) Record reproduction information for replay reproduction, policy rollback, or scheduling parameter update; E) Optionally, when the input-side constraint injection module is triggered, a safety constraint vector and / or safety cue words are injected into the input side of the main language model to affect subsequent output.