Large language model input-output pair risk discrimination method and device

Abstract

The application provides a large language model input-output pair risk discrimination method, comprising the following steps: decoupling the input-output pair into an input channel, an output channel and an interaction channel, and extracting initial features of the channels; vectorizing strategy text in a current strategy library to obtain a strategy embedding matrix, and fusing the strategy embedding matrix into the initial features of the three channels through a cross-attention mechanism to obtain strategy-enhanced features of the channels; splicing the strategy-enhanced features of the channels to obtain global fusion features, performing multi-dimensional quantification on uncertainty and controversy degrees of the global fusion features respectively, and constructing a risk discrimination condition; and screening high-uncertainty samples from a plurality of input-output pairs according to the risk discrimination condition and storing the high-uncertainty samples in a high-uncertainty sample set. The method can effectively discriminate whether the input-output pair formed by user input and model response is risky.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence security technology, and in particular to a method and apparatus for risk assessment of large language model input-output pairs. Background Technology

[0002] With the widespread application of Large Language Models (LLMs), the security of their generated content has become an increasing concern. How to effectively determine whether the input-output pairs consisting of user input and model response pose any risks has become a research hotspot in both academia and industry.

[0003] In existing technologies, auditing methods based on dedicated discriminative models primarily involve fine-tuning a pre-trained language model on a large amount of labeled security data. This enables the model to perform binary (safe / unsafe) or multi-class (specific risk category) classification on input-output pairs, or to provide natural language explanations along with the output results, illustrating the basis for the judgment and enhancing decision-making transparency. Taking the dedicated discriminative model Qwen3Guard as an example, two architectural variants are employed: a generative discriminative architecture, which reconstructs security classification into an instruction-following task, where the model receives explicit auditing instructions and security policies and outputs formatted classification results; and a streaming discriminative architecture, which adds a token-level classification head for real-time security monitoring during incremental generation. Methods based on general-purpose large language models as discriminators use carefully designed prompts to allow the general-purpose large language model to classify input-output pairs for risk and output risk categories and brief reasons. Methods based on rules and traditional classifiers use keyword matching, rule engines, or lightweight classifiers for fast discrimination, suitable for high-throughput, low-latency scenarios.

[0004] However, the aforementioned existing technical methods have the following problems:

[0005] 1. The "controversial" judgment in the three-classification scenario lacks interpretability.

[0006] While existing technologies introduce the "Controversial" category, their output is merely a label, failing to explain why a case is deemed controversial. In practice, many boundary cases require reviewers to understand the reasons for the controversy (such as cultural differences, scenario dependencies, or model uncertainty) in order to handle them correctly. Existing technologies cannot provide this information, leading to inefficient handling of controversial cases.

[0007] 2. Insufficient ability to identify input-output coupling risks

[0008] Existing methods often employ independent analysis of inputs and outputs, simply merging user inputs and model responses after separate risk assessments. However, real-world risks often lie in the interaction between inputs and outputs: user inputs may be harmless, but the model output may be harmful; inputs may be harmful, but the model correctly rejects them; or harmful inputs may generate harmful content in the output. Current technologies lack explicit modeling of these "input-output interactions," making it difficult to accurately identify such interactive risks.

[0009] 3. Security policies have poor dynamic adaptability and high update costs.

[0010] The existing discrimination model adopts a static training paradigm, and the model parameters are frozen after deployment. This design results in insufficient adaptability of the model when facing three types of changes: (1) new attack methods (such as new jailbreak templates and prompt injection patterns); (2) updates to security standards (such as the addition of new risk categories); and (3) cross-cultural deployment scenarios.

[0011] 4. Policy conflicts lead to performance instability.

[0012] Some existing technologies attempt to improve discrimination accuracy by introducing security policies, but the overall performance actually decreases after some policies are added; manually written policies may conflict with each other, causing the original policies to fail; policies have poor generalization ability, and policies that are effective in one scenario may be ineffective in another. Existing technologies lack a management mechanism for the relationships between policies, cannot automatically detect policy conflicts, and are difficult to effectively integrate policies into the discrimination model. Summary of the Invention

[0013] This invention provides a method and apparatus for risk assessment of large language model input-output pairs. Based on dynamic strategy generation and uncertainty quantification technology, it effectively assesses whether there is risk in the input-output pairs consisting of user input and model response.

[0014] This invention provides a method for risk assessment of large language model input-output pairs, comprising:

[0015] Several input-output pairs are fed into an online discriminant model, which includes a multi-channel encoder, a policy fusion module, and an uncertainty quantification and dispute resolution module; wherein, the user input text and the corresponding large language model response text constitute an input-output pair.

[0016] The multi-channel encoder decouples the input-output pair into three semantic channels: an input channel, an output channel, and an interaction channel, and extracts initial features for each channel. The input channel is used to analyze the malice level of the user input, the output channel is used to analyze the harmfulness level of the model response, and the interaction channel is used to explicitly calculate the semantic consistency and degree of malicious cooperation between the input and output.

[0017] The policy fusion module vectorizes the policy text of the current policy library to obtain a policy embedding matrix. The policy embedding matrix is ​​then fused into the initial features of the three channels through a cross-attention mechanism to obtain policy enhancement features for each channel.

[0018] The uncertainty quantification and dispute resolution module splices together the enhanced features of each channel strategy to obtain a global fusion feature. The global fusion feature is then used to quantify uncertainty and dispute level in multiple dimensions to construct risk judgment conditions.

[0019] Based on the risk discrimination criteria, high-uncertainty samples are selected from the several input-output pairs and stored in the high-uncertainty sample set.

[0020] In one embodiment of the present invention, the method further includes:

[0021] If the amount of data in the high uncertainty sample set reaches the trigger threshold, several high uncertainty samples are labeled to automatically generate candidate security strategies, thus obtaining a new strategy library.

[0022] The new policy library is compared with the current policy library to detect conflicts, and all candidate security policies that do not conflict are selected.

[0023] The online discrimination model is incrementally fine-tuned based on the labeled high-uncertainty samples and the conflict-free candidate security policies, and the online discrimination model and the current policy library are updated.

[0024] In one embodiment of the present invention, extracting the initial features of each channel includes:

[0025] The user input text and the response text of the large language model are respectively input into the pre-trained encoder, and the vector labeled [CLS] is taken as the sentence representation to obtain the initial features of the input channel and the initial features of the output channel.

[0026] The initial features of the input channel and the initial features of the output channel are concatenated to obtain the initial features of the interactive channel.

[0027] In one embodiment of the present invention, the method further includes: extracting the initial risk score for each channel, comprising:

[0028] The initial features of the input channel and the initial features of the output channel are respectively processed through a two-layer fully connected network to calculate the risk score, thereby obtaining the initial risk score of the input channel and the initial risk score of the output channel.

[0029] Using the initial features of the interaction channel, calculate the semantic alignment score and the malicious cooperation score;

[0030] The initial risk score of the interaction channel is obtained by multiplying the semantic alignment score and the malicious cooperation score.

[0031] In one embodiment of the present invention, the step of vectorizing the policy text of the current policy library to obtain a policy embedding matrix, and fusing the policy embedding matrix into the initial features of the three channels respectively through a cross-attention mechanism to obtain the policy enhancement features of each channel includes:

[0032] Encode each policy text in the current policy library to obtain a policy embedding vector, and stack all policy embedding vectors to obtain a policy embedding matrix;

[0033] Using the initial features of the input channel, the initial features of the output channel, and the initial features of the interaction channel as queries, and the policy embedding matrix as the key and value, the attention output is calculated to obtain the attention enhancement features of the input channel, the attention enhancement features of the output channel, and the attention enhancement features of the interaction channel.

[0034] For each channel, the initial features and attention-enhanced features are dynamically fused through a gating mechanism to obtain the policy-enhanced features for each channel.

[0035] In one embodiment of the present invention, the global fusion features are used to perform multi-dimensional quantification of both uncertainty and controversy, and risk discrimination conditions are constructed, including:

[0036] Uncertainty is quantified in two dimensions: accidental uncertainty and cognitive uncertainty.

[0037] The degree of controversy is quantified in three dimensions: cultural controversy, contextual controversy, and model controversy.

[0038] The global fusion features are input into the first prediction head to obtain a comprehensive risk score;

[0039] Enable the Dropout layer and perform at least one forward propagation to estimate cognitive uncertainty;

[0040] The global fusion features are input into the second prediction head to calculate the random uncertainty.

[0041] The total uncertainty is obtained by summing the cognitive uncertainty and the accidental uncertainty.

[0042] The global fusion features are input into the third, fourth, and fifth prediction heads to calculate the cultural controversy, contextual controversy, and model controversy, respectively.

[0043] Combining the comprehensive risk score and the total uncertainty, the risk discrimination criteria are constructed as follows:

[0044] If the overall risk score is greater than the preset high-risk threshold and the total uncertainty is less than the uncertainty threshold, output "risky";

[0045] If the overall risk score is less than or equal to the preset low-risk threshold and the total uncertainty is less than the uncertainty threshold, output "no risk";

[0046] If the total uncertainty is greater than or equal to the uncertainty threshold, the current input-output pair is identified as a high-uncertainty sample, the output is "disputed", and the dispute type is identified for the high-uncertainty sample;

[0047] The uncertainty quantification and dispute resolution module includes a first prediction head, a second prediction head, a third prediction head, a fourth prediction head, and a fifth prediction head.

[0048] In one embodiment of the present invention, enabling the Dropout layer and performing at least one forward propagation to estimate cognitive uncertainty includes:

[0049] The Dropout layer is enabled, and the same input-output pair is forward-propagated multiple times. The risk score of each forward-propagation output is recorded, and the cognitive uncertainty is calculated based on the dispersion of the multiple risk scores.

[0050] or,

[0051] The Dropout layer is enabled, and a single forward propagation is performed on the input-output pair to obtain the corresponding feature vector. The cognitive uncertainty is then calculated based on the feature variance of the feature vector.

[0052] In one embodiment of the present invention, the step of automatically generating candidate security strategies by labeling several high-uncertainty samples includes:

[0053] Select several samples from the high uncertainty sample set and label them manually or after screening using an active learning strategy;

[0054] For each labeled sample, the initial features of the input channel, the initial features of the output channel, and the initial features of the interaction channel are extracted. Then, a clustering algorithm is used to group the high-uncertainty samples to obtain multiple clusters.

[0055] For each cluster, select multiple samples, construct prompt words, and call a large language model to generate several strategy texts;

[0056] Calculate the policy confidence score of any policy text, and select policy texts that exceed the confidence score threshold as candidate security policy texts to add to the candidate policy list.

[0057] In one embodiment of the present invention, conflict detection between the new strategy library and the current strategy library includes:

[0058] A pre-trained semantic encoding model is used to encode the candidate security policies and all policy texts in the existing policy library respectively, so as to obtain the semantic embedding vectors corresponding to each policy.

[0059] The operation direction of each strategy is extracted based on the preset keyword matching rules: if the strategy text contains preset prohibited keywords, it is marked as a prohibited direction; if it contains preset allowed keywords, it is marked as an allowed direction; otherwise, it is determined as a neutral direction by a lightweight classification model.

[0060] A pre-defined semantic similarity measurement method is used to calculate the semantic similarity between the semantic embedding of the candidate security policy and the semantic embedding of each current policy;

[0061] Based on preset first and second similarity thresholds, and combined with the operational direction of the strategy, hierarchical discrimination is performed:

[0062] If the semantic similarity is less than the first preset similarity threshold, it is determined that there is no policy conflict;

[0063] If the semantic similarity is greater than or equal to the second preset similarity threshold and the operation direction is "prohibited direction", it is determined to be a direct conflict; if the semantic similarity is greater than or equal to the second preset similarity threshold and the operation direction is "allowed direction", it is determined to be a policy duplication.

[0064] If the semantic similarity is greater than or equal to the first preset similarity threshold and less than the second preset similarity threshold, and the operation direction is "prohibited direction", the natural language reasoning model is invoked to perform semantic contradiction verification. If the verification result is "contradictory", it is determined to be a potential conflict; otherwise, it is determined to be without conflict.

[0065] In one embodiment of the present invention, incremental fine-tuning of the online discrimination model based on labeled high-uncertainty samples and conflict-free candidate security strategies includes:

[0066] Deep copy the current parameters of the online discriminant model and the output distribution of the online discriminant model under the current policy base as a reference benchmark for knowledge distillation;

[0067] Before training begins, the conflict-free candidate security policies are added to the current policy library and encoded to obtain the corresponding embedding vectors.

[0068] Freeze all parameters of the multi-channel encoder except for a preset number of top-level encoding layers, train only the top-level encoding layer of the multi-channel encoder, the policy fusion module, and all prediction heads of the uncertainty quantization and dispute resolution module, and keep the policy embedding vectors of the policy library in a learnable state.

[0069] The labeled high-uncertainty samples are constructed as training batches. Each sample contains at least the user input text and the corresponding large language model response text, label, and sample weight. The sample weight is positively correlated with the total uncertainty of the sample.

[0070] Construct a total loss function, which contains three normalized components: weighted classification loss, knowledge distillation loss, and uncertainty regularization loss.

[0071] An adaptive weight decay optimizer is used, with a preset initial learning rate and a preset weight decay coefficient. The training strategy is configured as follows: a cosine annealing learning rate scheduling strategy is used to reduce the learning rate from the preset initial learning rate to the preset final learning rate; a gradient pruning mechanism is used to set a preset maximum gradient norm to prevent gradient explosion; and an early stopping mechanism is used to monitor the validation set loss and terminate training early when the preset threshold of no consecutive decreases is reached. At the same time, a preset maximum number of training rounds is set.

[0072] Incremental training is performed according to the configured training strategy. After each round of training, the performance is evaluated on the validation set, and the model parameters with the lowest validation loss are saved as the updated discriminant model parameters.

[0073] In one embodiment of the present invention, updating the online discrimination model and the current policy library includes:

[0074] The conflict-free candidate security policies are added to the persistent policy library, their embedding vectors are stored, and the policy library version number is updated to support hot reloading of online services.

[0075] First, write the updated online discrimination model parameters to the backup storage location, and then use an atomic pointer switching operation to point the online service to the updated online discrimination model, retaining the original model parameters for a preset duration to support rapid failure rollback;

[0076] Based on the validation set results after this incremental training, the uncertainty threshold for high-uncertainty samples is dynamically adjusted: if the overall uncertainty of the online discrimination model decreases after incremental training, the uncertainty threshold is increased to reduce the sampling frequency; if the proportion of high-uncertainty samples is still higher than the preset threshold after incremental training, the uncertainty threshold is decreased to increase the sampling frequency.

[0077] In one embodiment of the present invention, conflict detection between the new strategy library and the current strategy library includes:

[0078] The large language model is directly invoked to perform conflict detection between the new strategy library and the current strategy library.

[0079] Another aspect of the present invention provides a risk discrimination device for large language model input-output pairs, comprising:

[0080] Online discrimination unit, used for:

[0081] Several input-output pairs are fed into an online discriminant model, which includes a multi-channel encoder, a policy fusion module, and an uncertainty quantification and dispute resolution module; wherein, the user input text and the corresponding large language model response text constitute an input-output pair.

[0082] The multi-channel encoder decouples the input-output pair into three semantic channels: an input channel, an output channel, and an interaction channel, and extracts initial features for each channel. The input channel is used to analyze the malice level of the user input, the output channel is used to analyze the harmfulness level of the model response, and the interaction channel is used to explicitly calculate the semantic consistency and degree of malicious cooperation between the input and output.

[0083] The policy fusion module vectorizes the policy text of the current policy library to obtain a policy embedding matrix. The policy embedding matrix is ​​then fused into the initial features of the three channels through a cross-attention mechanism to obtain policy enhancement features for each channel.

[0084] The uncertainty quantification and dispute resolution module concatenates the enhancement features of each channel strategy to obtain a global fusion feature. The global fusion feature is then used to quantify uncertainty and dispute level in multiple dimensions to construct risk discrimination conditions. Based on the risk discrimination conditions, high uncertainty samples are selected from the input-output pairs and stored in the high uncertainty sample set.

[0085] Offline update unit, used for:

[0086] When the amount of data in the high-uncertainty sample set reaches the trigger threshold, several high-uncertainty samples are labeled to automatically generate candidate security policies, resulting in a new policy library. Conflict detection is performed between the new policy library and the current policy library to filter out all conflict-free candidate security policies. Based on the labeled high-uncertainty samples and the conflict-free candidate security policies, the online discrimination model is incrementally fine-tuned, and the online discrimination model and the current policy library are updated.

[0087] As can be seen from the above solutions, the advantages of the present invention are:

[0088] The large language model input-output pair risk discrimination method provided by this invention can effectively determine whether there is risk in the input-output pair consisting of user input and model response. Specifically, through multi-channel decoupling and explicit modeling of interaction risks, it can accurately identify "malicious input-cooperative output" type interaction risks, filling the blind spot of existing methods in the identification of input-output coupling risks. Through two-dimensional uncertainty quantification and hierarchical dispute analysis, "disputed" is upgraded from a single label to structured information including three dimensions: culture, context, and model, supporting users to understand the source of the dispute and take targeted measures, thus enhancing interpretability. Moreover, through automatic policy generation, conflict detection, and incremental learning mechanisms to update the discrimination model offline, the discrimination model can continuously adapt to new attack methods and security standard updates, alleviating the problem of risk surge in multi-turn dialogue scenarios. At the same time, the conflict detection mechanism ensures policy consistency and avoids performance degradation caused by contradictions between policies; policy vectorization fusion enables the model to truly understand human judgment standards. Policy and model inference are decoupled, and new policies can be implemented by updating the policy library and incremental learning, without requiring complete model retraining, which is expected to greatly reduce maintenance costs. Attached Figure Description

[0089] Figure 1 This is a schematic diagram of the overall process of the risk discrimination method for large language model input-output pairs provided in Embodiment 1 of the present invention;

[0090] Figure 2 This is a schematic diagram illustrating the working principle of the online discrimination model;

[0091] Figure 3 This is a schematic diagram of the overall process of the risk discrimination method for large language model input-output pairs provided in Embodiment 2 of the present invention;

[0092] Figure 4 This is a schematic diagram illustrating the overall working principle of online discrimination and offline updates.

[0093] Figure 5 This is an overall block diagram of the risk discrimination device for large language model input-output pairs provided in Embodiment 3 of the present invention.

[0094] The attached figures are labeled as follows:

[0095] 300: A risk discrimination device for large language model input-output pairs;

[0096] 310: Online discrimination unit;

[0097] 320: Offline update unit;

[0098] 11: Multi-channel encoder;

[0099] 12: Strategy Fusion Module;

[0100] 13: Uncertainty Quantification and Dispute Analysis Module. Detailed Implementation

[0101] It should be noted that, in this invention, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0102] In the absence of further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0103] In the research on risk discrimination methods for large model input-output pairs, it was found that the root cause of the above-mentioned defects in existing technologies lies in the fact that existing methods treat the discrimination model as a static black box and lack a systematic design for uncertainty, interaction relationships and strategy management. Specifically: (1) Existing technologies only output labels for the judgment of "controversial" because they regard controversy as a by-product of the discrimination result rather than a target that needs to be modeled. Controversy itself is multidimensional and may stem from cultural differences, scenario dependence or uncertainty of the model itself, and should be quantified separately. (2) Existing technologies ignore the risk of input-output interaction because they simplify the discrimination into a single-sample classification problem rather than a relationship modeling problem. The risk of input-output pairs depends not only on their respective attributes, but also on their semantic alignment and malicious cooperation relationship. (3) Existing technologies are difficult to dynamically adapt to new risks because they adopt a one-time "training-deployment" mode and cannot be updated incrementally. (4) Existing technologies have chaotic strategy management because the strategy exists in text form and is separated from the model. The strategy should be vectorized and integrated into the model, and a conflict detection mechanism should be established to ensure the consistency of the strategy.

[0104] In response, this invention provides a risk assessment method for large model input-output pairs based on dynamic strategy generation and uncertainty quantification. It achieves hierarchical dispute resolution, automatic strategy generation and conflict detection, and incremental updates driven by active learning through multi-channel decoupled modeling input-output interaction, dual-dimensional uncertainty quantification, and active learning-driven incremental updates.

[0105] Example 1

[0106] Specifically, refer to Figure 1 , Figure 2 As shown, Figure 1A schematic diagram of the overall process of a risk discrimination method for large language model input-output pairs provided in one embodiment of the present invention is provided. Figure 2 A schematic diagram illustrating the working principle of the online discrimination model is shown.

[0107] In this embodiment, an online discrimination model is constructed, comprising a multi-channel encoder 11, a policy fusion module 12, and an uncertainty quantification and dispute resolution module 13. In a specific implementation, a RoBERTa-base model is loaded as the backbone of the multi-channel encoder 11. The policy fusion module 12 can consist of a policy embedding layer, a cross-attention layer, and a gated fusion unit. The uncertainty quantification and dispute resolution module 13 can consist of a feature fusion layer and multiple independent binary classification prediction heads. The policy generation model can use the Qwen2.5-72B-Instruct model, initialized with an empty policy library. The online discrimination model is pre-trained on public security datasets such as WildGuardMix to enable basic discrimination capabilities.

[0108] A method for risk assessment of input-output pairs in a large language model, comprising the following steps:

[0109] Step S1: Input several input-output pairs into the constructed online discrimination model.

[0110] In this context, the user input text and the corresponding response text from the large language model constitute an input-output pair.

[0111] Step S2: Decouple the input-output pair into three semantic channels—input channel, output channel, and interaction channel—using the multi-channel encoder, and extract the initial features of each channel; wherein, the input channel is used to analyze the malice level of the user input, the output channel is used to analyze the harmfulness level of the model response, and the interaction channel is used to explicitly calculate the semantic consistency and malicious cooperation level between the input and output.

[0112] Step S3: Vectorize the policy text of the current policy library to obtain a policy embedding matrix through the policy fusion module, and fuse the policy embedding matrix into the initial features of the three channels through the cross-attention mechanism to obtain the policy enhancement features of each channel.

[0113] Step S4: The uncertainty quantification and dispute resolution module concatenates the enhancement features of each channel strategy to obtain a global fusion feature. This global fusion feature is then used to perform multi-dimensional quantification of both uncertainty and dispute level to construct risk discrimination conditions. Based on these risk discrimination conditions, high-uncertainty samples are selected from the input-output pairs and stored in a high-uncertainty sample set.

[0114] In this embodiment, the input-output pair is decoupled into three independent branches: an input channel, an output channel, and an interaction channel. The input channel analyzes the malice level of user input, the output channel analyzes the harmfulness level of the model response, and the interaction channel explicitly calculates the semantic consistency and malicious cooperation level between input and output to quantify interaction risks and identify the risk of malicious cooperation where "user input is harmful and model output is cooperative." By quantifying uncertainty and controversy in multiple dimensions, "controversial" is upgraded from a hard label to structured information, allowing users to understand the reasons for controversy (whether it is cultural difference, scenario dependence, or model uncertainty) and supporting differentiated processing.

[0115] In a preferred embodiment, in step S2, the user inputs text. Corresponding large language model response text Construct an input-output pair The user input text and the response text from the large language model are respectively input into the pre-trained encoder. The vector labeled [CLS] is taken as the sentence representation to obtain the initial features of the input channel. Initial characteristics of the output channel The pre-trained encoder, such as RoBERTa-base, has a hidden layer dimension of 768. Then, the initial features from the input channels are... Initial characteristics of the output channel By splicing, the initial features of the interactive channel are obtained. Simultaneously, the initial risk score for each channel is extracted; specifically, the initial features of the input channels are... Initial characteristics of the output channel The initial risk score for the input channel is obtained by calculating the risk score through two fully connected layers. Initial risk score for output channel The specific formula is:

[0116]

[0117]

[0118] in, This is the Sigmoid function, with an output range of [0,1]. , , , For the learnable parameters of the fully connected layer with input channels, , , , To output the learnable parameters of the fully connected layer in the output channel, It is a linear rectification activation function.

[0119] Utilizing the initial features of the interaction channel Calculate the semantic alignment score and the malicious cooperation score. The calculation formula is:

[0120]

[0121]

[0122] in, Indicates semantics regarding its fraction, To maliciously manipulate scores, , , , To align the learnable parameters of the score calculation layer, , , , Learnable parameters are available to complement the score calculation layer.

[0123] The initial risk score for the interaction channel is obtained by multiplying the semantic alignment score and the malicious cooperation score. ,Right now Initial risk score This reflects the definition of risk in interaction, which assumes that goal alignment and collaborative security are equally important, and that defects in either dimension will significantly increase the overall risk. Both Align and Collaborate take values ​​of [0,1].

[0124] In a preferred embodiment, step S3 involves vectorizing the policy texts in the current policy library to obtain a policy embedding matrix. The policy embedding matrix is ​​then fused into the initial features of the three channels using a cross-attention mechanism to obtain the policy enhancement features for each channel. Specifically, this includes: first, initializing the policy library, which stores multiple policies, each containing policy text and a corresponding embedding vector. For example, policy embeddings are encoded using Sentence-BERT (all-MiniLM-L6-v2 model, output dimension 384). Next, each policy text in the current policy library is encoded to obtain a policy embedding vector. All policy embedding vectors are then stacked to obtain a policy embedding matrix. ,in For the number of strategies, For the embedded dimension.

[0125] Using the initial features of the input channel, output channel, and interaction channel as queries, and the policy embedding matrix as keys and values, attention outputs are calculated to obtain attention-enhanced features for the input channel, output channel, and interaction channel. For each channel, the initial features and attention-enhanced features are dynamically fused using a gating mechanism to obtain the policy-enhanced features for each channel.

[0126] Specifically, based on the initial features of the input channel For query Strategy Embedding Matrix For key Sum Calculate the attention output to obtain the input channel attention enhancement features. :

[0127] ,in , , , It is a learnable projection matrix.

[0128] The initial features of the input channel are dynamically fused through a gating mechanism. Input channel attention enhancement features The input channel strategy enhancement features are obtained. : , .in To fuse the input channel threshold value, This is the element-wise product. , These are learnable parameters for the gating layer.

[0129] Similarly, the above process is repeated for the initial features of the output channel and the initial features of the interaction channel to obtain the output channel strategy enhancement features. Enhanced features of interaction channel strategy .

[0130] In a preferred embodiment, in step S4, the uncertainty quantification and dispute resolution module splices the enhancement features of each channel strategy to obtain a global fusion feature. The global fusion feature is then used to perform multi-dimensional quantification on both uncertainty and dispute level to construct risk discrimination conditions.

[0131] Specifically, uncertainty is quantified in two dimensions: accidental uncertainty and cognitive uncertainty; controversy is quantified in three dimensions: cultural controversy, contextual controversy, and model controversy. Table 1 explains the specific meaning of the three dimensions of controversy: "cultural controversy," "contextual controversy," and "model controversy."

[0132] Table 1. Meaning of Different Controversies

[0133]

[0134] The uncertainty quantification and dispute resolution module 13 can be composed of a feature fusion layer and multiple independent binary classification prediction heads, specifically including the first prediction head, the second prediction head, the third prediction head, the fourth prediction head, and the fifth prediction head.

[0135] Firstly, the three-channel strategy enhances features. By splicing the data, we obtain the global fusion features. , The globally fused features are input into the first prediction head to obtain a comprehensive risk score. , , This is a multilayer perceptron for risk prediction. A Dropout layer is enabled to perform at least one forward pass to estimate cognitive uncertainty. In a specific implementation, the Dropout layer can be enabled to perform multiple forward passes on the same input-output pair, for example... Second forward propagation ( (Take 10-20 passes), and record the risk score of each forward propagation output. The cognitive uncertainty is calculated based on the dispersion of multiple risk scores. ,Right now , This represents the average of multiple forward propagation risk scores. In another specific implementation, a Dropout layer is enabled to perform a single forward propagation on the input-output pair to obtain the corresponding feature vector. The cognitive uncertainty is then calculated based on the feature variance of this feature vector.

[0136] The global fusion features The data is input to a second prediction head, which directly outputs the inherent ambiguity of the data through this additional prediction head, and the random uncertainty is calculated. , Multilayer perceptron for predicting uncertainties.

[0137] The total uncertainty is obtained by summing the cognitive uncertainty and the accidental uncertainty. ,Right now .

[0138] The global fusion features are input into three independent prediction heads—the third, fourth, and fifth prediction heads—to calculate the dispute decomposition and obtain the cultural dispute degree for each. Contextual controversy Model Controversy ,Right now , , .

[0139] Preset high-risk threshold Preset low-risk threshold Preset uncertainty threshold The high-risk threshold is greater than the low-risk threshold, for example... , , .

[0140] Combining the comprehensive risk score and the total uncertainty, the risk discrimination criteria are constructed as follows:

[0141] If the comprehensive risk score Greater than the preset high-risk threshold And the total uncertainty Less than the uncertainty threshold ,Right now and Output "Risk";

[0142] If the comprehensive risk score Less than or equal to the preset low-risk threshold And the total uncertainty Less than the uncertainty threshold ,Right now and Output "No risk";

[0143] Otherwise, if the above two conditions are not met, it is considered "disputed," that is, if the total uncertainty is greater than or equal to the uncertainty threshold. The current input-output pair is identified as a high-uncertainty sample, and the output is "disputed". The type of dispute is then identified for each high-uncertainty sample, specifically the degree of cultural dispute. Contextual controversy Model Controversy The sample is compared with the corresponding threshold to further determine the dispute type of the high uncertainty sample.

[0144] Finally, based on the aforementioned risk discrimination criteria, high-uncertainty samples are selected from the several input-output pairs and stored in a high-uncertainty sample set. This high-uncertainty sample set... It includes several samples with high uncertainty (i.e., input-output pairs) and the corresponding comprehensive risk score for each sample. The total uncertainty mentioned above Information such as...

[0145] In this embodiment, through multi-channel decoupling and explicit modeling of interaction risks, the "malicious input-cooperative output" type of interaction risk can be accurately identified, filling the blind spot of existing methods in the identification of input-output coupling risks. By using two-dimensional uncertainty quantification and hierarchical dispute analysis, "controversial" is upgraded from a single label to structured information containing three dimensions: culture, context, and model. This supports users in understanding the source of the dispute and taking targeted measures, enhancing interpretability.

[0146] In addition, in one embodiment, for scenarios with limited computing resources, the following simplified solution can be adopted: replace the multi-channel encoder with DistilRoBERTa (66M parameters), store only policy text in the policy library, calculate the embedding online in real time, and sacrifice some latency.

[0147] In one embodiment, for scenarios requiring strict auditability, the policy fusion of the policy fusion module can be replaced with a rule-triggered mechanism: when the input-output pair matches a certain policy, the judgment result of that policy is directly applied to ensure the mandatory execution of key policies.

[0148] Example 2

[0149] refer to Figure 3 , Figure 4 As shown, Figure 3 This diagram illustrates the specific process of the risk assessment method based on the input-output pair of a large language model as disclosed in Embodiment 2. Figure 4 This is a schematic diagram illustrating the overall working principle of online identification and offline updates.

[0150] A method for risk assessment of input-output pairs in a large language model, comprising the following steps:

[0151] Step S1: Input several input-output pairs into the constructed online discriminant model, wherein the user input text and the corresponding large language model response text constitute an input-output pair.

[0152] Step S2: Decouple the input-output pair into three semantic channels—input channel, output channel, and interaction channel—using the multi-channel encoder, and extract the initial features of each channel; wherein, the input channel is used to analyze the malice level of the user input, the output channel is used to analyze the harmfulness level of the model response, and the interaction channel is used to explicitly calculate the semantic consistency and malicious cooperation level between the input and output.

[0153] Step S3: Vectorize the policy text of the current policy library to obtain a policy embedding matrix through the policy fusion module, and fuse the policy embedding matrix into the initial features of the three channels through the cross-attention mechanism to obtain the policy enhancement features of each channel.

[0154] Step S4: The uncertainty quantification and dispute resolution module concatenates the enhancement features of each channel strategy to obtain a global fusion feature. This global fusion feature is then used to perform multi-dimensional quantification of both uncertainty and dispute level to construct risk discrimination conditions. Based on these risk discrimination conditions, high-uncertainty samples are selected from the input-output pairs and stored in a high-uncertainty sample set.

[0155] Step S5: If the amount of data in the high uncertainty sample set reaches the trigger threshold, mark several high uncertainty samples to automatically generate candidate security policies and obtain a new policy library.

[0156] Step S6: Perform conflict detection between the new policy library and the current policy library, and filter out all candidate security policies that do not conflict.

[0157] Step S7: Based on the labeled high-uncertainty samples and the conflict-free candidate security policies, perform incremental fine-tuning on the online discrimination model and update the online discrimination model and the current policy library.

[0158] In this embodiment, an offline update stage is further added. Through automatic policy generation, conflict detection, and incremental learning mechanisms, the online discrimination model can continuously adapt to new attack methods and security standard updates, thereby alleviating the risk surge problem in multi-turn dialogue scenarios and improving the accuracy of effectively judging the existence of risks based on the input-output pair consisting of user input and model response.

[0159] In this embodiment, the relevant technical details mentioned in the above embodiments are still valid for steps S1 to S4, and will not be repeated here to reduce repetition.

[0160] In a preferred embodiment, in step S5, if the amount of data in the high-uncertainty sample set reaches a trigger threshold (e.g., 1000 samples), an offline update trigger process is initiated, and several high-uncertainty samples are labeled to automatically generate candidate security policies, resulting in a new policy library. Specifically, several samples in the high-uncertainty sample set are selected for manual labeling or labeled after screening using an active learning strategy. Initial features of the input channel are extracted for each labeled sample. Initial characteristics of the output channel Initial features of the interaction channel Clustering algorithms are used to group high-uncertainty samples into multiple clusters. For example, the DBSCAN clustering algorithm can be used with a radius parameter of 0.5 and a minimum number of samples of 3.

[0161] For each cluster, select multiple samples (up to 10 samples) and construct prompt words. The construction of prompt words is as follows:

[0162] "You are a content security strategy expert. Below is a set of high-uncertainty samples (input-output pairs that the model struggles to determine as violations):"

[0163] Sample 1: - User input: {X1}; - Model response: {Y1}; - Current risk score: {R1}; - Uncertainty: {U1};

[0164] Sample 2:...

[0165] Please analyze the common risk patterns of this batch of samples and generate a new content security strategy.

[0166] Output format: Common feature analysis: Risk pattern summary: New strategy: (using explicit expressions such as "allow / prohibit / restrict")

[0167] Then, the large language model is called to generate several policy texts based on the prompt words. In a specific implementation, an open-source model (such as Qwen2.5-72B-Instruct) or an API (such as GPT-4) can be used, and a temperature parameter of 0.3 can be set to ensure output stability.

[0168] Furthermore, the policy confidence score of any policy text is calculated, and policy texts exceeding the confidence threshold are selected as candidate security policy texts. Add to the candidate strategy list. Confidence = Number of clustered samples / Total number of high-uncertainty samples.

[0169] In this embodiment, features are extracted from high-uncertainty samples and cluster analysis is performed. Then, a large language model is used to automatically extract common risk patterns and generate candidate security strategies. The strategies are described in natural language to ensure readability and executability. No manual coding is required; new strategies are automatically extracted from boundary cases, reducing strategy maintenance costs and improving the model's adaptability to new risk scenarios. The two-stage design of "clustering first, then extraction" aims to avoid the problems of context length limitations, inductive bias, and poor reproducibility faced by large language models when directly inductively summarizing. Cluster analysis objectively discovers the inherent grouping structure of samples based on the feature space, compressing large-scale samples into several clusters; the large language model only needs to interpret patterns from representative samples within each cluster, thereby generating fine-grained and highly readable security strategies.

[0170] In a preferred embodiment, in step S6, conflict detection is performed between the new policy library and the current policy library to filter out all candidate security policies that do not conflict.

[0171] Specifically, firstly, a pre-trained semantic encoding model (such as Sentence-BERT) is used to encode the candidate security policy and all policy texts in the existing policy library to obtain the semantic embedding vectors corresponding to each policy.

[0172] The operation direction of each strategy is extracted based on preset keyword matching rules: if the strategy text contains preset prohibited keywords, it is marked as a prohibited direction; if it contains preset allowed keywords, it is marked as an allowed direction; otherwise, it is determined as a neutral direction by a lightweight classification model.

[0173] In a specific implementation, the preset prohibited keywords are such as "prohibited," "must not," "not allowed," "strictly prohibited," etc., and the preset permitted keywords are such as "allowed," "may," "encourage," "support," etc., but are not limited thereto. The prohibited direction can be marked as -1, the permitted direction can be marked as 1, and the neutral direction can be marked as 0, but the present invention is not limited thereto.

[0174] Furthermore, a pre-defined semantic similarity measurement method is used to calculate the semantic similarity between the semantic embedding of the candidate security policy and the semantic embedding of each current policy. For example, the pre-defined semantic similarity measurement method can use cosine similarity to represent semantic similarity.

[0175] Based on preset first and second similarity thresholds, and combined with the operational direction of the strategy, hierarchical discrimination is performed:

[0176] If the semantic similarity is less than the first preset similarity threshold, it is determined that there is no policy conflict;

[0177] If the semantic similarity is greater than or equal to the second preset similarity threshold and the operation direction is "prohibited direction", it is determined to be a direct conflict; if the semantic similarity is greater than or equal to the second preset similarity threshold and the operation direction is "allowed direction", it is determined to be a policy duplication.

[0178] If the semantic similarity is greater than or equal to the first preset similarity threshold and less than the second preset similarity threshold, and the operation direction is "prohibited direction", a natural language inference model (e.g., using microsoft / deberta-v3-base-mnli) is invoked to perform semantic contradiction verification. If the verification result is "contradictory", it is determined to be a potential conflict; otherwise, it is determined to be without conflict.

[0179] Where the first similarity threshold is less than the second similarity threshold, the first similarity threshold is set. Second similarity threshold .

[0180] The final output is a list of conflict-free strategies. In addition, all conflict judgment results are summarized to generate a list of conflict strategies, including conflict strategy, conflict type, similarity, and judgment reason.

[0181] In a specific implementation, the types of conflict can be further divided into several types, such as direct opposition, scope-inclusive conflict, conditional conflict, value conflict, regional conflict, and scenario conflict. The specific definitions of each type of conflict are shown in Table 2.

[0182] Table 2 lists conflict boundary cases.

[0183]

[0184] In this embodiment, the policy text is converted into vectors using a semantic encoding model. The semantic similarity between the new policy and the existing policy library is calculated, and the policy's operational direction (allow / prohibit / neutral) is extracted. A natural language inference model is then used to determine if a genuine semantic contradiction exists. Three scenarios are distinguished: direct conflict, potential conflict, and duplication, each triggering a different processing flow. This ensures that the newly added policy does not conflict with the current policy library, avoiding performance degradation caused by policy contradictions. Potential conflicts trigger manual review to guarantee the quality of the policy library.

[0185] In a preferred embodiment, in step S7, based on the labeled high uncertainty samples... The online discrimination model is incrementally fine-tuned using the conflict-free candidate security policies, and the online discrimination model and the current policy library are updated.

[0186] Specifically, the current parameters of the online discriminant model are first deeply copied for subsequent knowledge distillation loss calculation. Simultaneously, the output distribution of the online discriminant model under the current policy library is saved as a reference benchmark for knowledge distillation.

[0187] Before training begins, the conflict-free candidate security policies are added to the current policy library and encoded into corresponding embedding vectors. The policy fusion module then uses a cross-modal attention mechanism to integrate these policy vectors into the feature representation of the multi-channel discriminative model. A gating mechanism dynamically fuses the initial features and policy enhancement features. Through policy vectorization fusion, the human judgment criteria are truly internalized into the model's decision-making process, enabling the model to genuinely understand these criteria. This allows the model to learn the semantic representation of new policies during incremental training, achieving joint optimization of the policy and the model.

[0188] Then, most parameters are frozen, and a parameter-efficient fine-tuning strategy is adopted to reduce the risk of overfitting: specifically, all parameters of the multi-channel encoder except for a preset number of top-level encoding layers are frozen, and only the top-level encoding layer of the multi-channel encoder, the policy fusion module, and all prediction heads of the uncertainty quantization and dispute resolution module are trained. Furthermore, the policy embedding vectors in the policy library are kept in a learnable state to allow for fine-tuning.

[0189] Preparing Training Data: High-uncertainty labeled samples are constructed as training batches. Each sample contains at least the user input text, the corresponding large language model response text, labeled tags, and sample weights. The sample weights are positively correlated with the total uncertainty of the sample. For example, labeled tags... (0 = safe, 1 = controversial, 2 = unsafe), sample weights ,in This is the weight scaling factor, which defaults to 2.0.

[0190] Construct a total loss function, which includes three normalized components: weighted classification loss, knowledge distillation loss, and uncertainty regularization loss.

[0191] Weighted classification loss To introduce sample weights into the cross-entropy loss, it is expressed as:

[0192]

[0193] in For the sample Category The logit output, For the first Each sample corresponds to its true category. The logit output.

[0194] Knowledge distillation loss Using KL divergence ensures that the output distribution of the new model on old data approximates that of the old model, preventing catastrophic forgetting. This is represented as:

[0195]

[0196] in and These represent the predicted probability distributions of the updated discriminant model and the initial discriminant model on the initial samples, respectively.

[0197] Uncertainty Regularization Loss To encourage models to make more certain predictions for samples with high uncertainty, while reducing uncertainty for samples that are already certain, it is expressed as:

[0198]

[0199] in, () represents the indicator function, and δ represents the error tolerance threshold. This is the predicted risk score output by the model for the i-th sample. Assign a true risk value to the i-th sample. Apply uncertainty penalty only to samples with incorrect or ambiguous predictions to avoid blindly compressing the uncertainty of all samples.

[0200] Total loss function

[0201] hyperparameters , , Determined through a grid search of the validation set, typical values ​​are... , , The three loss components are normalized before training to ensure they are of the same order of magnitude.

[0202] Then, incremental training is performed. Specifically, an adaptive weight decay optimizer is used, with preset initial learning rate and preset weight decay coefficients. Training strategies are configured as follows: a cosine annealing learning rate scheduling strategy is used to reduce the learning rate from the preset initial learning rate to the preset final learning rate; a gradient pruning mechanism is used to set a preset maximum gradient norm to prevent gradient explosion; and an early stopping mechanism is used to monitor the validation set loss, terminating training early when a preset threshold of no consecutive decreases is reached, while also setting a preset maximum number of training epochs. Incremental training is performed according to the configured training strategy. After each training epoch, the performance is evaluated on the validation set, and the model parameters with the lowest validation loss are saved as the updated discriminant model parameters.

[0203] In a specific implementation, the AdamW optimizer is used, with a learning rate of... Weight decay is 0.01. Training strategy: Number of training epochs: 3; Learning rate scheduling: cosine annealing, initial learning rate... Final learning rate Gradient clipping: Maximum gradient norm is 1.0 to prevent gradient explosion; Early stopping mechanism: Monitor the loss on the validation set, and terminate early if the loss does not decrease for two consecutive epochs.

[0204] The conflict-free candidate security policies are added to the persistent policy library. Here, persistence is only confirmed, and their embedding vectors are stored. The policy library version number is then updated to support hot reloading of online services.

[0205] A double-buffering mechanism is further adopted to achieve hot updates: first, the updated online discrimination model parameters are written to a backup storage location, and then the online service is pointed to the updated online discrimination model through an atomic pointer switching operation, while the original model parameters are retained for a preset duration to support rapid failure rollback.

[0206] Based on the validation set results after this incremental training, the uncertainty threshold for high-uncertainty samples is dynamically adjusted: if the overall uncertainty of the online discrimination model decreases after incremental training, the uncertainty threshold is increased to reduce the sampling frequency; if the proportion of high-uncertainty samples is still higher than the preset threshold after incremental training, the uncertainty threshold is decreased to increase the sampling frequency.

[0207] In this embodiment, during the online discrimination process, samples with uncertainty exceeding a threshold are stored in a buffer pool. When the buffer pool is full, an offline update process is triggered: candidate strategies are generated based on the high-uncertainty samples in the buffer pool; conflict detection is performed on the candidate strategies, filtering out entries that conflict with or duplicate existing strategies; and conflict-free strategies are combined with high-uncertainty samples to incrementally fine-tune the discrimination model. This forms a continuous optimization closed loop of "uncertainty screening → strategy generation → conflict detection → incremental update," enabling the model to dynamically adapt to new risk patterns through incremental learning, without manual intervention or complete retraining, significantly reducing maintenance costs.

[0208] Furthermore, in one embodiment, conflict detection between the new strategy library and the current strategy library can be performed by directly calling a large language model.

[0209] For scenarios requiring stronger interpretability, the main model can be replaced with a generative large language model (such as Qwen2.5-7B-Instruct), and end-to-end discrimination can be achieved through fine-tuning of instructions. However, the inference latency is relatively high, making it suitable for non-real-time scenarios.

[0210] Example 3:

[0211] refer to Figure 5 As shown, Figure 5 A schematic diagram of the overall structure of a risk discrimination device based on the input-output pair of a large language model is shown.

[0212] A risk discrimination device 300 for input-output pairs of a large language model includes:

[0213] Online discrimination unit 310, used for:

[0214] Several input-output pairs are fed into an online discriminant model, which includes a multi-channel encoder, a policy fusion module, and an uncertainty quantification and dispute resolution module; wherein, the user input text and the corresponding large language model response text constitute an input-output pair.

[0215] The multi-channel encoder decouples the input-output pair into three semantic channels: an input channel, an output channel, and an interaction channel, and extracts initial features for each channel. The input channel is used to analyze the malice level of the user input, the output channel is used to analyze the harmfulness level of the model response, and the interaction channel is used to explicitly calculate the semantic consistency and degree of malicious cooperation between the input and output.

[0216] The policy fusion module vectorizes the policy text of the current policy library to obtain a policy embedding matrix. The policy embedding matrix is ​​then fused into the initial features of the three channels through a cross-attention mechanism to obtain policy enhancement features for each channel.

[0217] The uncertainty quantification and dispute resolution module concatenates the enhancement features of each channel strategy to obtain a global fusion feature. The global fusion feature is then used to quantify uncertainty and dispute in multiple dimensions to construct risk discrimination conditions. Based on the risk discrimination conditions, high uncertainty samples are selected from the input-output pairs and stored in the high uncertainty sample set.

[0218] Offline update unit 320, used for:

[0219] When the amount of data in the high-uncertainty sample set reaches the trigger threshold, several high-uncertainty samples are labeled to automatically generate candidate security policies, resulting in a new policy library. Conflict detection is performed between the new policy library and the current policy library to filter out all conflict-free candidate security policies. Based on the labeled high-uncertainty samples and the conflict-free candidate security policies, the online discrimination model is incrementally fine-tuned, and the online discrimination model and the current policy library are updated.

[0220] It should be noted that this device implementation can be combined with the above-described method implementation. The relevant technical details mentioned in the above method implementation remain valid in this device implementation, and will not be repeated here to avoid repetition.

[0221] In addition, embodiments of the present invention also provide a readable storage medium storing a program or instructions, which, when executed by a processor, implements the steps of the above-described large language model input-output risk discrimination method and achieves the same technical effect.

[0222] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described large language model input-output risk discrimination method and achieves the same technical effect.

[0223] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

Claims

1. A method for risk assessment of input-output pairs in a large language model, characterized in that, include: Several input-output pairs are fed into an online discriminant model, which includes a multi-channel encoder, a policy fusion module, and an uncertainty quantification and dispute resolution module; wherein, the user input text and the corresponding large language model response text constitute an input-output pair. The multi-channel encoder decouples the input-output pair into three semantic channels: an input channel, an output channel, and an interaction channel, and extracts initial features for each channel. The input channel is used to analyze the malice level of the user input, the output channel is used to analyze the harmfulness level of the model response, and the interaction channel is used to explicitly calculate the semantic consistency and degree of malicious cooperation between the input and output. The policy fusion module vectorizes the policy text of the current policy library to obtain a policy embedding matrix. The policy embedding matrix is ​​then fused into the initial features of the three channels through a cross-attention mechanism to obtain policy enhancement features for each channel. The uncertainty quantification and dispute resolution module splices together the enhanced features of each channel strategy to obtain a global fusion feature. The global fusion feature is then used to quantify uncertainty and dispute level in multiple dimensions to construct risk judgment conditions. Based on the risk discrimination criteria, high-uncertainty samples are selected from the several input-output pairs and stored in the high-uncertainty sample set.

2. The method according to claim 1, characterized in that, Also includes: If the amount of data in the high uncertainty sample set reaches the trigger threshold, several high uncertainty samples are labeled to automatically generate candidate security strategies, thus obtaining a new strategy library. The new policy library is compared with the current policy library to detect conflicts, and all candidate security policies that do not conflict are selected. The online discrimination model is incrementally fine-tuned based on the labeled high-uncertainty samples and the conflict-free candidate security policies, and the online discrimination model and the current policy library are updated.

3. The method according to claim 1, characterized in that, Extract the initial features of each channel, including: The user input text and the response text of the large language model are respectively input into the pre-trained encoder, and the vector labeled [CLS] is taken as the sentence representation to obtain the initial features of the input channel and the initial features of the output channel. The initial features of the input channel and the initial features of the output channel are concatenated to obtain the initial features of the interactive channel.

4. The method according to claim 3, characterized in that, It also includes: extracting the initial risk score for each channel, including: The initial features of the input channel and the initial features of the output channel are respectively processed through a two-layer fully connected network to calculate the risk score, thereby obtaining the initial risk score of the input channel and the initial risk score of the output channel. Using the initial features of the interaction channel, calculate the semantic alignment score and the malicious cooperation score; The initial risk score of the interaction channel is obtained by multiplying the semantic alignment score and the malicious cooperation score.

5. The method according to claim 1, characterized in that, The steps of vectorizing the policy text of the current policy library to obtain a policy embedding matrix, and fusing the policy embedding matrix into the initial features of the three channels through a cross-attention mechanism to obtain the policy enhancement features of each channel include: Encode each policy text in the current policy library to obtain a policy embedding vector, and stack all policy embedding vectors to obtain a policy embedding matrix; Using the initial features of the input channel, the initial features of the output channel, and the initial features of the interaction channel as queries, and the policy embedding matrix as the key and value, the attention output is calculated to obtain the attention enhancement features of the input channel, the attention enhancement features of the output channel, and the attention enhancement features of the interaction channel. For each channel, the initial features and attention-enhanced features are dynamically fused through a gating mechanism to obtain the policy-enhanced features for each channel.

6. The method according to claim 1, characterized in that, By utilizing the aforementioned global fusion features, multi-dimensional quantification is performed on both uncertainty and controversy levels to construct risk discrimination conditions, including: Uncertainty is quantified in two dimensions: accidental uncertainty and cognitive uncertainty. The degree of controversy is quantified in three dimensions: cultural controversy, contextual controversy, and model controversy. The global fusion features are input into the first prediction head to obtain a comprehensive risk score; Enable the Dropout layer and perform at least one forward propagation to estimate cognitive uncertainty; The global fusion features are input into the second prediction head to calculate the random uncertainty. The total uncertainty is obtained by summing the cognitive uncertainty and the accidental uncertainty. The global fusion features are input into the third, fourth, and fifth prediction heads to calculate the cultural controversy, contextual controversy, and model controversy, respectively. Combining the comprehensive risk score and the total uncertainty, the risk discrimination criteria are constructed as follows: If the overall risk score is greater than the preset high-risk threshold and the total uncertainty is less than the uncertainty threshold, output "Riskful"; If the overall risk score is less than or equal to the preset low-risk threshold and the total uncertainty is less than the uncertainty threshold, output "no risk"; If the total uncertainty is greater than or equal to the uncertainty threshold, the current input-output pair is identified as a high-uncertainty sample, the output is "disputed", and the dispute type is identified for the high-uncertainty sample; The uncertainty quantification and dispute resolution module includes a first prediction head, a second prediction head, a third prediction head, a fourth prediction head, and a fifth prediction head.

7. The method according to claim 1, characterized in that, Enable the Dropout layer, perform at least one forward propagation, and estimate cognitive uncertainty, including: The Dropout layer is enabled, and the same input-output pair is forward-propagated multiple times. The risk score of each forward-propagation output is recorded, and the cognitive uncertainty is calculated based on the dispersion of the multiple risk scores. or, The Dropout layer is enabled, and a single forward propagation is performed on the input-output pair to obtain the corresponding feature vector. The cognitive uncertainty is then calculated based on the feature variance of the feature vector.

8. The method according to claim 2, characterized in that, The automatic generation of candidate security strategies from labeled high-uncertainty samples includes: Select several samples from the high uncertainty sample set and label them manually or after screening using an active learning strategy; For each labeled sample, the initial features of the input channel, the initial features of the output channel, and the initial features of the interaction channel are extracted. Then, a clustering algorithm is used to group the high-uncertainty samples to obtain multiple clusters. For each cluster, select multiple samples, construct prompt words, and call a large language model to generate several strategy texts; Calculate the policy confidence score of any policy text, and select policy texts that exceed the confidence score threshold as candidate security policy texts to add to the candidate policy list.

9. The method according to claim 8, characterized in that, Conflict detection between the new policy library and the current policy library includes: A pre-trained semantic encoding model is used to encode the candidate security policies and all policy texts in the existing policy library respectively, so as to obtain the semantic embedding vectors corresponding to each policy. The operation direction of each strategy is extracted based on the preset keyword matching rules: if the strategy text contains preset prohibited keywords, it is marked as a prohibited direction; if it contains preset allowed keywords, it is marked as an allowed direction; otherwise, it is determined as a neutral direction by a lightweight classification model. A pre-defined semantic similarity measurement method is used to calculate the semantic similarity between the semantic embedding of the candidate security policy and the semantic embedding of each current policy; Based on preset first and second similarity thresholds, and combined with the operational direction of the strategy, hierarchical discrimination is performed: If the semantic similarity is less than the first preset similarity threshold, it is determined that there is no policy conflict; If the semantic similarity is greater than or equal to the second preset similarity threshold and the operation direction is "prohibited direction", then it is determined to be a direct conflict; If the semantic similarity is greater than or equal to the second preset similarity threshold and the operation direction is "allowed direction", it is determined as a policy duplication; If the semantic similarity is greater than or equal to the first preset similarity threshold and less than the second preset similarity threshold, and the operation direction is "prohibited direction", the natural language reasoning model is invoked to perform semantic contradiction verification. If the verification result is "contradictory", it is determined to be a potential conflict; otherwise, it is determined to be without conflict.

10. The method according to claim 2, characterized in that, Incremental fine-tuning of the online discrimination model based on labeled high-uncertainty samples and conflict-free candidate security strategies includes: Deep copy the current parameters of the online discriminant model and the output distribution of the online discriminant model under the current policy base as a reference benchmark for knowledge distillation; Before training begins, the conflict-free candidate security policies are added to the current policy library and encoded to obtain the corresponding embedding vectors. Freeze all parameters of the multi-channel encoder except for a preset number of top-level encoding layers, train only the top-level encoding layer of the multi-channel encoder, the policy fusion module, and all prediction heads of the uncertainty quantization and dispute resolution module, and keep the policy embedding vectors of the policy library in a learnable state. The labeled high-uncertainty samples are constructed as training batches. Each sample contains at least the user input text and the corresponding large language model response text, label, and sample weight. The sample weight is positively correlated with the total uncertainty of the sample. Construct a total loss function, which contains three normalized components: weighted classification loss, knowledge distillation loss, and uncertainty regularization loss. An adaptive weight decay optimizer is used, with a preset initial learning rate and a preset weight decay coefficient. The training strategy is configured as follows: a cosine annealing learning rate scheduling strategy is used to reduce the learning rate from the preset initial learning rate to the preset final learning rate; a gradient pruning mechanism is used to set a preset maximum gradient norm to prevent gradient explosion; and an early stopping mechanism is used to monitor the validation set loss and terminate training early when the preset threshold of no consecutive decreases is reached. At the same time, a preset maximum number of training rounds is set. Incremental training is performed according to the configured training strategy. After each round of training, the performance is evaluated on the validation set, and the model parameters with the lowest validation loss are saved as the updated discriminant model parameters.

11. The method according to claim 10, characterized in that, Updating the online discrimination model and the current policy library includes: The conflict-free candidate security policies are added to the persistent policy library, their embedding vectors are stored, and the policy library version number is updated to support hot reloading of online services. First, write the updated online discrimination model parameters to the backup storage location, and then use an atomic pointer switching operation to point the online service to the updated online discrimination model, retaining the original model parameters for a preset duration to support rapid failure rollback; Based on the validation set results after this incremental training, the uncertainty threshold for high-uncertainty samples is dynamically adjusted: if the overall uncertainty of the online discrimination model decreases after incremental training, the uncertainty threshold is increased to reduce the sampling frequency; if the proportion of high-uncertainty samples is still higher than the preset threshold after incremental training, the uncertainty threshold is decreased to increase the sampling frequency.

12. The method according to claim 2, characterized in that, Conflict detection between the new policy library and the current policy library includes: The large language model is directly invoked to perform conflict detection between the new strategy library and the current strategy library.

13. A risk discrimination device for input-output pairs of a large language model, characterized in that, include: Online discrimination unit, used for: Several input-output pairs are fed into an online discriminant model, which includes a multi-channel encoder, a policy fusion module, and an uncertainty quantification and dispute resolution module; wherein, the user input text and the corresponding large language model response text constitute an input-output pair. The multi-channel encoder decouples the input-output pair into three semantic channels: an input channel, an output channel, and an interaction channel, and extracts initial features for each channel. The input channel is used to analyze the malice level of the user input, the output channel is used to analyze the harmfulness level of the model response, and the interaction channel is used to explicitly calculate the semantic consistency and degree of malicious cooperation between the input and output. The policy fusion module vectorizes the policy text of the current policy library to obtain a policy embedding matrix. The policy embedding matrix is ​​then fused into the initial features of the three channels through a cross-attention mechanism to obtain policy enhancement features for each channel. The uncertainty quantification and dispute resolution module concatenates the enhancement features of each channel strategy to obtain a global fusion feature. The global fusion feature is then used to quantify uncertainty and dispute level in multiple dimensions to construct risk discrimination conditions. Based on the risk discrimination conditions, high uncertainty samples are selected from the input-output pairs and stored in the high uncertainty sample set. Offline update unit, used for: When the amount of data in the high-uncertainty sample set reaches the trigger threshold, several high-uncertainty samples are labeled to automatically generate candidate security policies, resulting in a new policy library. Conflict detection is performed between the new policy library and the current policy library to filter out all conflict-free candidate security policies. Based on the labeled high-uncertainty samples and the conflict-free candidate security policies, the online discrimination model is incrementally fine-tuned, and the online discrimination model and the current policy library are updated.

Images