An Adaptive Quantification Method for Hazardous Knowledge Risk in Multimodal Systems Based on Cross-Modal Risk Origin Tracing and Risk Consistency Calibration

Abstract

This invention relates to an adaptive quantification method for harmful knowledge risk within a multimodal context, based on cross-modal risk tracing and risk consistency calibration. This invention can locate risk parameters related to harmful knowledge, improving the targeted nature of model correction. It can quantify risk by combining candidate response difference features and risk semantic stability, thereby reducing the probability of low-confidence anomalous outputs being judged as high-risk samples and triggering parameter updates. Furthermore, this invention only updates risk-related local parameters, which helps maintain the model's original general capabilities.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence security technology, and in particular to an adaptive quantification method for harmful knowledge risk in multimodal contexts based on cross-modal risk tracing and risk consistency calibration. Background Technology

[0002] Multimodal large models possess strong capabilities in image understanding, text generation, and cross-modal reasoning tasks. However, during pre-training, instruction fine-tuning, or subsequent alignment, they may absorb and store knowledge related to violence, hate speech, discrimination, illegal activities, or other unsafe content. When specific images, text prompts, or combinations of both meet certain triggering conditions, the model may output responses with harmful biases. This risk does not entirely stem from the individual input sample itself but may be related to the parameterized knowledge stored within the model.

[0003] Most existing security control methods start from the final output, detecting the harmfulness of the model's answers and controlling rejection or fine-tuning the overall security. These methods typically have the following problems: First, judging solely based on the final output makes it difficult to distinguish whether harmful content is caused by harmful knowledge stably stored within the model, or by decoding fluctuations, semantic uncertainty, or occasional shifts. Second, in multimodal scenarios, risk triggering paths often span multiple stages, including visual representation, modal mapping, and language generation; without localizing the internal parameter carriers, effective correction is difficult. Third, directly fine-tuning or forgetting training the entire model can easily lead to a simultaneous degradation of general knowledge, language ability, and visual understanding, resulting in catastrophic forgetting.

[0004] Therefore, it is necessary to propose a technical solution that can simultaneously complete risk parameter positioning, risk level quantification, and local constraint correction, so as to improve the pertinence, stability, and controllability of processing endogenous harmful knowledge in multimodal large models. Summary of the Invention

[0005] To address the challenges in existing technologies regarding accurately determining the parameter-carrying location of harmful knowledge within a large multimodal model, uniformly and stably quantifying the risk level under different input triggers, and minimizing damage to the generality of the large multimodal model while correcting it, this invention provides an adaptive quantification method for harmful knowledge risk within a multimodal context based on cross-modal risk tracing and risk consistency calibration, comprising the following steps: S1. Obtain the input samples of the multimodal large model to be detected and corrected, generate a candidate response set and extract intermediate representations; S2. Perform cross-modal risk tracing to determine the risk parameter set, and expand the risk parameter set according to the parameter gradient similarity to obtain the expanded risk parameter set; S3. Perform risk consistency calibration, determine the risk direction consistency of candidate responses based on the difference in comparison probability between harmful candidate responses and safe candidate responses, determine the risk semantic stability based on the risk semantic entropy obtained from several sampling decodings, and calculate the risk quantification value based on the risk direction consistency and risk semantic stability. S4. Classify the input samples according to the risk quantification value to obtain high-risk sample set, medium-risk sample set and low-risk sample set; S5. For high-risk sample sets, generate parameter masks based on extended risk parameter sets, and perform local parameter updates under parameter mask constraints and capability preservation constraints. S6. Re-execute S3 on the updated model for retesting and verification, and evaluate whether the model's general capability offset meets the preset safety conditions.

[0006] Preferably, S2 includes: Based on the probability of harmful candidate responses corresponding to multimodal input samples, a coarse screening is performed on the candidate attention heads and feedforward channels in the large language model layer; Risk response interventions were performed on the candidate attention heads and feedforward channel activations after coarse screening, and the risk source contribution value of the candidate components was determined based on the change in the probability of harmful candidate responses before and after the intervention. If the contribution value of risk tracing is higher than the threshold, the corresponding parameter will be included in the risk parameter set.

[0007] Preferably, the risk tracing contribution value is calculated according to the following formula: In the formula, Indicates the first Layer The risk tracing contribution value corresponding to each candidate component. Indicates the input sample Expectation calculation; This indicates that the model is based on the input samples. Generating harmful candidate responses under certain conditions The probability of; Indicates the first Layer Activation values ​​of candidate components After setting it to zero, the model will input samples Generating harmful candidate responses under certain conditions The probability of; This indicates that the candidate component is intervened in and its activation is disabled.

[0008] Preferably, S2 further includes: When parameter With parameters correlation between When the value exceeds the preset threshold, the parameter will be... Add the aforementioned extended risk parameter set; Relevance Calculate according to the following formula: in, Representing parameters respectively and parameters The gradient vector on the risk sample.

[0009] Preferably, S3 includes: Harmful candidate responses and safe candidate responses are selected from the candidate response set, the difference in comparison probability is calculated, and the difference in comparison probability is used as a consistency indicator of the risk direction of candidate responses; The same input sample is randomly sampled and decoded multiple times to obtain several semantic clusters and their probabilities. The risk semantic entropy is calculated and used as a calibration index for the stability of risk semantics. By substituting the difference in comparative probabilities and the semantic entropy of risk into the normalized mapping function, a comprehensive risk quantification value is obtained.

[0010] Preferably, the difference in comparison probability is calculated according to the following formula: In the formula, Indicates input sample The difference in the probability of comparison This indicates that the model is based on the input samples. Generating harmful candidate responses under certain conditions The probability, This indicates that the model is based on the input samples. Generate safe candidate responses under certain conditions The probability, Indicates the smoothing term; Risk semantic entropy is calculated according to the following formula: In the formula, Indicates input sample Risk semantic entropy, Indicates the first The probability corresponding to each semantic cluster Indicates the number of semantic clusters; The risk quantification value is calculated using the following formula: In the formula, Indicates input sample The risk quantification value, Represents the normalized mapping function. This represents the weighting parameter.

[0011] Preferably, S4 includes: When the risk quantification value is greater than or equal to the first threshold, the corresponding sample is added to the high-risk sample set; When the risk quantification value is less than the first threshold and greater than or equal to the second threshold, the corresponding sample is added to the medium-risk sample set, and inference period security control is performed on the samples in the medium-risk sample set. When the risk quantification value is less than the second threshold, the corresponding sample is added to the low-risk sample set.

[0012] Preferably, S5 includes: Parameter updates are performed based on a joint optimization objective function, which is a weighted sum of risk mitigation loss, capability preservation loss, distribution constraints with the reference model, and parameter offset constraints.

[0013] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention can locate risk parameters related to harmful knowledge and improve the pertinence of model correction.

[0014] (2) The present invention can combine the candidate response difference features and risk semantic stability to quantify risk, thereby reducing the probability that low confidence abnormal outputs are judged as high-risk samples and trigger parameter updates.

[0015] (3) This invention only updates the risk-related local parameters, which helps to maintain the original general capabilities of the model. Attached Figure Description

[0016] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of an adaptive quantification method for harmful knowledge risk within a multimodal context based on cross-modal risk tracing and risk consistency calibration, according to an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] Example 1 like Figure 1 The diagram shown is a schematic representation of the method flow in this embodiment, and the steps include: S1. Obtain the input samples of the multimodal large model to be detected and corrected, generate a candidate response set and extract intermediate representations.

[0021] This embodiment uses a multimodal large model as an example to illustrate the steps of the method of the present invention. The selected model includes a visual encoder, a modal projection module, and a large language model backbone, and the input samples are denoted as... ,in Indicates image input, This indicates text input.

[0022] Input the image and text input Input a large multimodal model to obtain a set of candidate responses. The intermediate representation for risk tracing is extracted, which includes candidate attention head output and candidate feedforward channel activation in the large language model layer after image-text fusion.

[0023] S2. Perform cross-modal risk sourcing to determine the risk parameter set, and expand the risk parameter set according to the parameter gradient similarity to obtain the extended risk parameter set.

[0024] First, based on the harmful candidate response probabilities corresponding to the multimodal input samples, candidate attention heads and feedforward channels in the large language model layer are coarsely screened. Then, risk response interventions are performed on the activations of the coarsely screened candidate attention heads and feedforward channels, and the risk tracing contribution value of the candidate components is determined based on the change in harmful candidate response probabilities before and after the intervention. For the... Layer The risk tracing contribution value of each candidate component is calculated according to formula (1).

[0025] In the formula, Indicates the first Layer The risk tracing contribution value corresponding to each candidate component. Indicates the input sample The expected operation. This indicates that the model is based on the input samples. Generating harmful candidate responses under certain conditions The probability of. Indicates the first Layer Activation values ​​of candidate components After setting it to zero, the model will input samples Generating harmful candidate responses under certain conditions The probability of. This indicates that the candidate component is intervened in and its activation is disabled.

[0026] If the result obtained by formula (1) is Above the threshold Then the parameters corresponding to the candidate component are included in the risk parameter set. To reduce the likelihood of models reconstructing harmful content using related knowledge, parameter gradient similarity is also applied. To extend this, the gradient similarity is calculated according to equation (2).

[0027] In equation (2), Indicates parameters With parameters The degree of correlation between them. and Representing parameters respectively and parameters The gradient vector on the risk sample. This represents the cosine similarity. This represents the vector dot product. and Representing vectors respectively and The norm of .

[0028] when Greater than the preset threshold When, the parameters Add to the extended risk parameter set The resulting parameter set includes both the physical carriers that directly affect harmful outputs and related parameters that are highly consistent with their update direction.

[0029] S3. Perform risk consistency calibration, determine the risk direction consistency of candidate responses based on the difference in comparison probability between harmful candidate responses and safe candidate responses, determine the risk semantic stability based on the risk semantic entropy obtained from several sampling decodings, and calculate the risk quantification value based on the risk direction consistency and risk semantic stability.

[0030] Calculate the risk quantification value. For the input sample... First, start with the candidate response set. Selecting harmful candidate responses and safe candidate responses Then, the difference in contrast probability is calculated according to equation (3). .

[0031] In equation (3), Indicates input sample The difference in the probability of comparison This indicates that the model is based on the input samples. Generating harmful candidate responses under certain conditions The probability, This indicates that the model is based on the input samples. Generate safe candidate responses under certain conditions The probability, This indicates the smoothing term.

[0032] Next, the same input sample was processed. After random sampling and decoding, several semantic clusters and their probabilities are obtained. Then calculate the risk semantic entropy according to equation (4). .

[0033] In equation (4), Indicates input sample Risk semantic entropy, Indicates the first The probability corresponding to each semantic cluster Indicates the number of semantic clusters.

[0034] Finally, the probability differences will be compared. and risk semantic entropy Substituting into equation (5) yields the comprehensive risk quantification value. .

[0035] In equation (5), Indicates input sample The risk quantification value, Represents the normalized mapping function. This represents the weighting parameter.

[0036] like A larger value indicates that the model tends to generate harmful candidate responses more strongly than safe candidate responses. If... The lower value indicates that the results of several samplings are concentrated in similar risk semantic clusters, and the model has relatively stable internal support for the current harmful semantics.

[0037] S4. Based on the risk quantification value, the input samples are classified into high-risk, medium-risk, and low-risk sample sets.

[0038] Implement tiered processing. Let the first threshold be... The second threshold is ,and .when At that time, the sample was added to the high-risk sample set. ;when At that time, the sample will be added to the medium-risk sample set. ;when At that time, the sample was added to the low-risk sample set. .for For medium-risk samples, instead of directly updating the model parameters, the key regions of the image are converted into corresponding text descriptions and input together with the original text prompts to output a safe response.

[0039] S5. For high-risk sample sets, generate parameter masks based on extended risk parameter sets, and perform local parameter updates under parameter mask constraints and capability preservation constraints.

[0040] Perform mask-constrained local parameter updates. This is for high-risk sample sets. Training sample pairs consisting of harmful and safe responses are constructed, and based on an expanded set of risk parameters... Generate parameter mask Local parameter updates are performed according to equation (6).

[0041] In equation (6), Indicates the update step size. Indicates the parameter mask. Represents element-wise product. Indicates the first Model parameters during round update, Indicates the first The updated model parameters, Describe the joint optimization objective function Regarding parameters The gradient.

[0042] Among them, only the mask Parameters with a value of 1 are allowed to be updated. The joint optimization objective function is constructed according to equation (7): In equation (7), This represents the joint optimization objective function. This indicates that risk mitigates loss. This indicates a loss of ability to maintain. This represents the distribution constraint term between the reference model and the model. Indicates the parameter mask. This indicates the parameter offset constraint term.

[0043] S6. Re-execute S3 on the updated model for retesting and verification, and evaluate whether the model's general capability offset meets the preset safety conditions.

[0044] Perform a retest and validation. After updating the local parameters, re-execute S3 on the updated model, comparing the changes in the risk source contribution value and risk quantification value of the risk parameter set. Simultaneously, evaluate the model performance shift on general visual question answering, image and text understanding, and text generation samples. If the risk quantification value decreases and the general capability shift does not exceed a preset threshold, output the updated model; otherwise, adjust the updated parameters and re-execute from S2.

[0045] Example 2 To verify the effectiveness of the method of this invention, in this embodiment, LLaVA-1.5-7B is used as the base model, the publicly available multimodal security evaluation dataset MM-SafetyBench is selected as the security evaluation dataset, and MMBEC, ScienceQA-IMG, and TextVQA are selected as general capability preservation evaluation datasets. Specifically, MM-SafetyBench is used to evaluate the model's insecurity response rate in multiple types of image-text risk scenarios, MMBEC is used to evaluate the model's general image-text understanding capability, ScienceQA-IMG is used to evaluate the model's multimodal reasoning capability, and TextVQA is used to evaluate the model's image-text reading and question-answering capability. The original LLaVA-1.5-7B, ECSO, ETA, and the method of this invention are compared. The results are shown in Table 1.

[0046] Table 1: Comparison of the previous methods on different datasets As shown in Table 1, in the comparative experiments based on the LLaVA-1.5-7B model, the insecure response rate of the method of this invention on MM-SafetyBench is significantly lower than that of other comparative methods, indicating that the present invention has a better effect in suppressing multimodal harmful responses. Meanwhile, the metrics of the method of this invention on MMBEC, ScienceQA-IMG, and TextVQA show only slight fluctuations compared to the original LLaVA-1.5-7B, and remain at a similar level compared to ECSO and ETA, indicating that while reducing the insecure response rate, it has little impact on the model's original general text-image understanding, multimodal question answering, and image-text reading capabilities.

[0047] In summary, this invention can effectively locate, quantify, and correct the inherent harmful knowledge in multimodal large models, while maintaining both the safety correction effect and the general capability, thus verifying the feasibility and effectiveness of the method of this invention.

[0048] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. An adaptive quantification method for harmful knowledge risk within multiple modalities based on cross-modal risk tracing and risk consistency calibration, characterized in that, include: S1. Obtain the input samples of the multimodal large model to be detected and corrected, generate a candidate response set and extract intermediate representations; S2. Perform cross-modal risk tracing, determine the risk parameter set, and expand the risk parameter set according to the parameter gradient similarity to obtain the expanded risk parameter set; S3. Perform risk consistency calibration, determine the risk direction consistency of candidate responses based on the difference in comparison probability between harmful candidate responses and safe candidate responses, determine the risk semantic stability based on the risk semantic entropy obtained from several sampling decodings, and calculate the risk quantification value based on the risk direction consistency and risk semantic stability. S4. Classify the input samples according to the risk quantification value to obtain high-risk sample set, medium-risk sample set and low-risk sample set; S5. For high-risk sample sets, generate parameter masks based on extended risk parameter sets, and perform local parameter updates under parameter mask constraints and capability preservation constraints. S6. Re-execute S3 on the updated model for retesting and verification, and evaluate whether the model's general capability offset meets the preset safety conditions.

2. The adaptive quantification method for harmful knowledge risk within multiple modalities based on cross-modal risk tracing and risk consistency calibration as described in claim 1, characterized in that, S2 includes: Based on the probability of harmful candidate responses corresponding to multimodal input samples, a coarse screening is performed on the candidate attention heads and feedforward channels in the large language model layer; Risk response interventions were performed on the candidate attention heads and feedforward channel activations after coarse screening, and the risk source contribution value of the candidate components was determined based on the change in the probability of harmful candidate responses before and after the intervention. If the contribution value of risk tracing is higher than the threshold, the corresponding parameter will be included in the risk parameter set.

3. The adaptive quantification method for harmful knowledge risk within multiple modalities based on cross-modal risk tracing and risk consistency calibration as described in claim 2, is characterized in that... The contribution value for risk tracing is calculated using the following formula: In the formula, Indicates the first Layer The risk tracing contribution value corresponding to each candidate component. Indicates the input sample Expectation calculation; This indicates that the model is based on the input samples. Generating harmful candidate responses under certain conditions The probability of; Indicates the first Layer Activation values ​​of candidate components After setting it to zero, the model will input samples Generating harmful candidate responses under certain conditions The probability of; This indicates that the candidate component is intervened in and its activation is disabled.

4. The adaptive quantification method for harmful knowledge risk within multiple modalities based on cross-modal risk tracing and risk consistency calibration as described in claim 1, characterized in that, S2 further includes: When parameter With parameters correlation between When the value exceeds the preset threshold, the parameter will be... Add the aforementioned extended risk parameter set; Relevance Calculate according to the following formula: in, Representing parameters respectively and parameters The gradient vector on the risk sample.

5. The adaptive quantification method for harmful knowledge risk within multiple modalities based on cross-modal risk tracing and risk consistency calibration as described in claim 1, characterized in that, S3 includes: Harmful candidate responses and safe candidate responses are selected from the candidate response set, the difference in comparison probability is calculated, and the difference in comparison probability is used as a consistency indicator of the risk direction of candidate responses; The same input sample is randomly sampled and decoded multiple times to obtain several semantic clusters and their probabilities. The risk semantic entropy is calculated and used as a calibration index for the stability of risk semantics. By substituting the difference in comparative probabilities and the semantic entropy of risk into the normalized mapping function, a comprehensive risk quantification value is obtained.

6. The adaptive quantification method for harmful knowledge risk within multiple modalities based on cross-modal risk tracing and risk consistency calibration as described in claim 5, is characterized in that... The difference in comparison probabilities is calculated according to the following formula: In the formula, Indicates input sample The difference in the probability of comparison This indicates that the model is based on the input samples. Generating harmful candidate responses under certain conditions The probability, This indicates that the model is based on the input samples. Generate safe candidate responses under certain conditions The probability, Indicates the smoothing term; Risk semantic entropy is calculated according to the following formula: In the formula, Indicates input sample Risk semantic entropy, Indicates the first The probability corresponding to each semantic cluster Indicates the number of semantic clusters; The risk quantification value is calculated according to the following formula: In the formula, Indicates input sample The risk quantification value, Represents the normalized mapping function. This represents the weighting parameter.

7. The adaptive quantification method for harmful knowledge risk within multiple modalities based on cross-modal risk tracing and risk consistency calibration as described in claim 1, characterized in that, S4 includes: When the risk quantification value is greater than or equal to the first threshold, the corresponding sample is added to the high-risk sample set; When the risk quantification value is less than the first threshold and greater than or equal to the second threshold, the corresponding sample is added to the medium-risk sample set, and inference period security control is performed on the samples in the medium-risk sample set. When the risk quantification value is less than the second threshold, the corresponding sample is added to the low-risk sample set.

8. The method according to claim 1, characterized in that, S5 includes: Parameter updates are performed based on a joint optimization objective function, which is a weighted sum of risk mitigation loss, capability preservation loss, distribution constraints with the reference model, and parameter offset constraints.

Images