Abnormal output management method and system for realizing large language model based on SIA architecture
By using the SISA architecture to shard training data and train large language sub-models, and calculating anomaly scores, the problems of implicit semantic leakage and difficulty in tracing responsibility in large language models are solved, achieving efficient security management and responsibility identification.
Patent Information
- Application Number
- CN202511085663.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-04
- Publication Date
- 2025-11-18
AI Technical Summary
Large language models pose a risk of implicit semantic leakage in data combinatorial reasoning, and lack a high-precision accountability and auditing path, making it difficult to attribute security responsibilities. Traditional methods are also unable to identify and prevent implicit sensitive content generated by combinatorial reasoning.
The SISA architecture is used to divide the training data into multiple subsets, and large language sub-models are trained separately. By calculating the risk confidence of sensitive words and text similarity of candidate answers, anomaly scores are calculated to identify suspicious training samples and establish an interpretable mapping path from abnormal model behavior to training data.
It achieves high-precision identification and interception of combined semantic risks, reduces model repair costs, provides fine-grained data security management and accountability capabilities, and supports security compliance and accountability.
Smart Images
Figure CN120974484A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of model management technology, and in particular to a method and system for managing the abnormal output of large language models based on the SISA architecture. Background Technology
[0002] In today's era of rapid technological advancement, Large Language Models (LLMs) have been widely applied in numerous key scenarios such as intelligent question answering and decision support due to their powerful language understanding and generation capabilities. However, with the deepening of their application, security and controllability issues have become increasingly prominent, becoming key factors restricting their further development and secure application. These issues are mainly manifested in the following aspects:
[0003] 1. Data-level security risks
[0004] Under the data-driven retraining method, the semantic risk of the model output does not depend entirely on whether a single piece of training data is sensitive, but rather on the semantic leakage risk brought about by the combined reasoning between multiple data fragments.
[0005] In practical applications, training data is often rigorously screened and de-identified, theoretically containing no explicit personal privacy or confidential information. For example, in the medical field, explicit sensitive information such as patients' names and ID numbers is removed. However, large language models possess powerful pattern combination and logical reasoning capabilities, enabling them to extract information from seemingly unrelated data fragments and perform combinatorial reasoning. Even if the input information itself is not sensitive, the model may synthesize potentially private content or sensitive inferences in the output. For instance, it can infer specific disease information from seemingly ordinary data combinations such as a patient's consultation time, symptom description, and medication records. Because traditional methods primarily process explicit sensitive fields in single samples, they cannot address this cross-sample, deep-level semantic association and combinatorial leakage. Furthermore, since this combinatorial reasoning leakage is based on the combinatorial reasoning of multiple data fragments, it is not only extremely difficult to prevent using traditional de-identification methods, but also difficult to trace its source at the level of a single training sample, posing a significant challenge to data security.
[0006] Therefore, some attackers exploit this characteristic of large language models by using data poisoning to maliciously attack the models. They carefully design malicious samples that do not explicitly represent confidential content or harmful topics and upload them to the database. Because these malicious samples do not contain obvious sensitive information on the surface, they often escape the data cleaning process and are submitted to the large language model for training. During training, these malicious samples induce the model to generate errors or sensitive content, thereby manipulating the model's output or causing implicit leaks. For example, attackers can mix seemingly normal text into the training data, but hide specific patterns or logic within it. When the model learns these patterns, it will generate unexpected or even harmful outputs when faced with specific inputs. This data poisoning behavior not only affects the model's performance and accuracy but also poses a serious threat to user privacy and data security.
[0007] 2. Deficiencies in the audit accountability mechanism
[0008] In the face of the aforementioned synthetic semantic leakage or combination-triggered behaviors that occur in large language models, there is currently a lack of highly accurate accountability and auditing paths, making it difficult to attribute security responsibilities.
[0009] Traditional data governance processes primarily rely on anonymizing training data, such as removing personal information and contextual fragments, supplemented by manual review to filter potentially sensitive content. However, anonymization methods can only eliminate explicit sensitive fields and are insufficient for handling deep semantic relationships or cross-sample combination leaks. Furthermore, once data passes through the screening process and enters the training process, effective tracing and accountability become virtually impossible later. For example, when sensitive information is leaked in the model output, current alignment training methods mostly start from the input and output but lack a mapping path from training data fragments to model behavior. Consequently, it is impossible to accurately trace which samples or combinations of samples triggered the abnormal semantics, thus failing to explain why the model generated certain abnormal responses. This results in a lack of evidence to support security audits and attribution of responsibility.
[0010] To address these shortcomings, some methods attempt to implement filtering mechanisms at the model output stage, such as through security policies, blacklists, and RLHF (Reinforcement Learning from Human Feedback) to control output behavior. However, these methods cannot trace the source of the problem. They struggle to provide a valid chain of evidence for security audits, regulatory compliance, and liability determination. For example, while blacklists can prevent the model from outputting words on the blacklist, they cannot identify or block implicitly sensitive content generated through combinatorial reasoning. Furthermore, because it's impossible to determine how this sensitive content was formed during model training, sufficient evidence cannot be provided to prove the model's compliance and security when facing security audits and regulatory compliance requirements. Summary of the Invention
[0011] To address the problems existing in the prior art, this invention provides a method for managing the abnormal output of a large language model based on the SISA architecture, comprising:
[0012] Step S1: Divide the training data into multiple subsets, and train a large language sub-model for each subset.
[0013] Step S2: The user's question is sent to each of the large language sub-models simultaneously to obtain the candidate answers and corresponding model output confidence scores of each large language model.
[0014] Step S3: Calculate the sensitive word risk confidence of each candidate answer, and the text similarity between each candidate answer and the remaining candidate answers;
[0015] Step S4: Calculate the average confidence level of each of the sensitive word risk confidence levels and the average similarity level of each of the text similarities.
[0016] Step S5: Calculate the anomaly score corresponding to each candidate answer based on the model output confidence, the sensitive word risk confidence, the average confidence and the average similarity, and determine the candidate answer with the anomaly score greater than the anomaly threshold as an abnormal output, and then lock the subset associated with the large language sub-model corresponding to the candidate answer as a suspicious training sample.
[0017] Preferably, in step S3, calculating the confidence level of sensitive word risk for each candidate answer includes:
[0018] Step S31: Count the total number of times each sensitive word in the candidate answer appears in the preset sensitive word list;
[0019] Step S32: Calculate the proportion of sensitive word characters in the candidate answer based on the total number of occurrences, the text length of each sensitive word appearing in the candidate answer, and the text length of the candidate answer.
[0020] Step S33: Calculate the ratio of the number of each of the sensitive words appearing in the candidate answer to the text length of the candidate answer, and perform a weighted summation of the ratio and the weight to obtain the sensitive word risk score;
[0021] Step S34: Adjust the model output confidence score based on the sensitive word risk score to obtain the sensitive word risk confidence score.
[0022] Preferably, the formula for calculating the specific gravity is as follows:
[0023]
[0024] Among them, R sens (y i ) represents the specific gravity, y i Let S = {s1, s2, ..., s} represent the candidate answers generated by the large language sub-model i. K} represents the list of sensitive words, Length(s) k ) indicates sensitive word s k The length of the text, Count(s) k ,y i ) indicates sensitive word s k In candidate answer y i The number of times L appears in i Indicates candidate answer y i The length of the text.
[0025] Preferably, the formula for calculating the risk confidence of the sensitive words is as follows:
[0026]
[0027] Among them, R(y) i ) represents the sensitive word risk score of the candidate answer corresponding to the large language sub-model i. c represents the confidence level of the model output corresponding to the large language sub-model i. i The risk confidence of the sensitive words in the candidate answer corresponding to the large language sub-model i is represented.
[0028] Preferably, the formula for calculating the anomaly score is as follows:
[0029]
[0030] Among them, A iThe anomaly score represents the score with weights w1, w2, w3 ≥ 0, and w1 + w2 + w3 = 1, κ ≥ 0, and τ, s th ∈(0,1), c represents the average confidence level. i Represents the large language submodel i pairs
[0031] The corresponding confidence level of the sensitive word risk in the candidate answer. R(y) represents the average similarity. i ) represents the sensitive word risk score of the candidate answer corresponding to the large language sub-model i.
[0032] Preferably, in step S5, after locking the suspicious training sample, a local repair operation is performed, which includes retraining or fine-tuning the corresponding large language sub-model, and then returning to step S2.
[0033] Preferably, the method further includes continuously generating anomaly detection and local repair operation logs during the anomaly detection process of steps S1 to S5, and binding the corresponding suspicious training samples and the corresponding large language sub-model for subsequent security audits and accountability tracing.
[0034] This invention also provides an exception output management system for large language models based on the SISA architecture. Applying the above-described exception output management method, the exception output management system includes:
[0035] The data sharding and training module is used to divide the training data into multiple subsets and train a large language sub-model for each subset.
[0036] The output generation module, connected to the data sharding and training module, is used to simultaneously send user questions to each of the large language sub-models to obtain candidate answers and corresponding model output confidence scores for each of the large language models.
[0037] An anomaly detection module, connected to the output generation module, is used to calculate the sensitive word risk confidence of each candidate answer and the text similarity between each candidate answer and the remaining candidate answers;
[0038] The calculation module, connected to the anomaly detection module, is used to calculate the average confidence of the risk confidence of each of the sensitive words and the average similarity of each of the text similarities;
[0039] The source tracing and localization module is connected to the output generation module, the anomaly detection module, and the calculation module, respectively. It is used to calculate the anomaly score corresponding to each candidate answer based on the model output confidence, the sensitive word risk confidence, the average confidence, and the average similarity. The candidate answer with an anomaly score greater than the anomaly threshold is judged as an abnormal output, and then the subset associated with the large language sub-model corresponding to the candidate answer is locked as a suspicious training sample.
[0040] Preferably, it also includes a local repair module connected to the source tracing and localization module, used to perform a local repair operation after locking the suspicious training sample, the local repair operation including retraining or fine-tuning the corresponding large language sub-model.
[0041] Preferably, it also includes an audit and accountability module, which is connected to the source tracing and localization module and the local repair module respectively. It is used to continuously generate anomaly detection and local repair operation logs during the anomaly detection process, and bind the corresponding suspicious training samples and the corresponding large language sub-model for subsequent security audit and accountability tracing.
[0042] The above technical solution has the following advantages or beneficial effects:
[0043] 1) By constructing a training and auditing mechanism based on the SISA architecture, a subset of training data is bound to the output of the large language sub-model obtained from the training, thereby enabling the deduction of suspicious training samples that triggered the anomaly from the abnormal output.
[0044] 2) By aggregating and analyzing candidate answers output by multiple large language sub-models, and calculating the anomaly score of each candidate answer based on the model output confidence, sensitive word risk confidence, and text similarity among all candidate answers, we can achieve high-precision identification and interception of implicit combination attacks from the output consistency and confidence fluctuation indicators of large language sub-models, especially the implicit semantic triggering of "single point harmless, multi-point combination harmful", and enhance the ability to perceive the combined semantic risks.
[0045] 3) Detect abnormal outputs by comparing anomaly scores with anomaly thresholds, and inversely infer suspicious training samples associated with abnormal outputs. Establish an interpretable mapping path from abnormal model behavior to training data, realize accurate location and source tracing of poisoning data and its impact, and thus reveal deep semantic chains and security risks triggered by cross-sample combinations.
[0046] 4) After detecting an abnormal output of a certain large language sub-model, it is only necessary to perform negative sample retraining or model replacement for the corresponding data subset, without having to retrain all data and models, thereby significantly reducing repair costs, improving intervention efficiency, and ensuring rapid model recovery and stable operation.
[0047] 5) Based on the binding relationship between training data shards and sub-model outputs, a full-link traceable path from abnormal semantic output to training data fragments is realized, supporting fine-grained data security management, auditing and responsibility division, and providing strong technical support for security compliance and accountability. Attached Figure Description
[0048] Figure 1 A flowchart illustrating a method for managing the abnormal output of a large language model based on the SISA architecture, as a preferred embodiment of the present invention.
[0049] Figure 2 This is a schematic diagram of a sub-process for calculating the risk confidence of sensitive words for each candidate answer in a preferred embodiment of the present invention.
[0050] Figure 3 This is a schematic diagram of the structure of an exception output management system for a large language model based on the SISA architecture, which is a preferred embodiment of the present invention. Detailed Implementation
[0051] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. The present invention is not limited to this embodiment; other embodiments that conform to the spirit of the present invention may also fall within the scope of the present invention.
[0052] In a preferred embodiment of the present invention, based on the above-mentioned problems existing in the prior art, a method for managing the abnormal output of a large language model based on the SISA architecture is provided, such as... Figure 1 As shown, it includes:
[0053] Step S1: Divide the training data into multiple subsets and train a large language sub-model for each subset.
[0054] Step S2: Send the user's question to each of the major language sub-models simultaneously to obtain the candidate answers and corresponding model output confidence scores of each major language model.
[0055] Step S3: Calculate the risk confidence of sensitive words for each candidate answer, and the text similarity between each candidate answer and the other candidate answers;
[0056] Step S4: Calculate the average confidence level of the risk confidence level of each sensitive word and the average similarity level of each text similarity.
[0057] Step S5: Calculate the anomaly score for each candidate answer based on the model output confidence, sensitive word risk confidence, average confidence and average similarity. Then, determine the candidate answers with anomaly scores greater than the anomaly threshold as abnormal outputs, and lock the subset associated with the large language sub-model corresponding to the candidate answers as suspicious training samples.
[0058] Specifically, to address the problems of difficulty in locating abnormal behavior, difficulty in identifying combined semantic leakage, high model repair costs, and lack of data accountability in existing technologies, this invention adopts a training and auditing mechanism based on the SISA (Sharded, Isolated, Sliced, and Aggregated) architecture. The specific principle is as follows:
[0059] 1. Data partitioning and independent model training
[0060] The training data is divided into multiple independent subsets (Shards), and a corresponding large language sub-model is trained separately for each subset (Shard). This ensures that the training process is isolated from each other, avoids data cross-contamination, and achieves a one-to-one binding relationship between the training data and the output of the large language sub-model.
[0061] Each large language sub-model uses only its corresponding subset of data for training and inference, achieving "isolation" and "slicing" in the training process. This ensures that the output of each large language sub-model is bound to its training data, thereby achieving fine-grained data behavior association.
[0062] This binding mechanism ensures a traceable path from abnormal model output to training data fragments, providing a solid technical foundation for security auditing and accountability.
[0063] 2. Sub-model output aggregation and consistency analysis
[0064] All major language sub-models are run in parallel to generate candidate answers and corresponding model output confidence scores for each input request. The prediction results of each major language sub-model are then aggregated. Abnormal output behavior is detected by comparing the consistency and confidence fluctuation of the outputs of each major language sub-model.
[0065] Specifically, the text similarity between each candidate answer and the other candidate answers is calculated, and then the average similarity of each text similarity is calculated as a consistency index to characterize the consistency between the outputs of each large language sub-model.
[0066] By calculating the confidence level of the sensitive word risk for each candidate answer, and then calculating the average confidence level of each sensitive word risk as a confidence fluctuation index, the confidence fluctuation of each large language sub-model is characterized.
[0067] Based on these metrics, by calculating the anomaly score corresponding to each candidate answer and judging candidate answers with anomaly scores greater than the anomaly threshold as abnormal outputs, the abnormal patterns of model outputs can be identified in a timely manner, especially the implicit semantic triggers of "harmless on a single point, harmful in combination of multiple points", thereby enhancing the ability to perceive combined semantic risks.
[0068] This method breaks through the limitations of traditional single-sample detection, possesses the ability to identify complex combined attacks with high accuracy, and significantly improves the model's security awareness and risk prevention level.
[0069] 3. Mechanism for tracing the source of poisoning effects
[0070] Because each large language sub-model is trained independently during data sharding, when a candidate answer is judged as an anomalous output, the specific data fragment behind the anomalous output can be identified through reverse inference. This establishes an interpretable mapping path from the model's abnormal behavior to the training data, enabling precise location and tracing of the poisoned data and its impact. This mechanism can reveal deep semantic chains and security risks triggered by cross-sample combinations.
[0071] Based on the binding relationship between abnormal output and sub-model training logs, the system reverse-engineers and accurately locates the training data shards and specific samples that caused the abnormal output, achieving a transparent mapping from model behavior to training data. This tracing mechanism fills the gap in existing technologies where abnormal behavior is untraceable, providing a complete and interpretable semantic triggering chain for abnormalities, effectively supporting security incident investigations and liability determination.
[0072] 4. Local model repair and rapid intervention
[0073] After detecting an anomaly in the output of a large language sub-model, only negative sample retraining or model replacement is required for the corresponding data subset. There is no need to retrain all data and the model, which significantly reduces the repair cost, improves the intervention efficiency, and ensures the rapid recovery and stable operation of the model.
[0074] 5. Full-chain traceability of safety responsibilities
[0075] Construct an audit log system covering the entire process of anomaly detection, source tracing, and repair, recording key data such as training data, sub-model outputs, and repair operations, forming a complete safety responsibility traceability chain.
[0076] Based on the binding relationship between training data fragments and the output of large language sub-models, a traceable path is achieved across the entire chain from abnormal semantic output to training data fragments. This supports fine-grained data security management, auditing, and accountability, providing strong technical support for security compliance and accountability. Furthermore, this auditing system meets regulatory compliance and security management requirements, ensuring transparency and controllability throughout the data security process, and providing legal and technical protection for user rights and organizational security.
[0077] The following specific examples illustrate the execution process of the above-mentioned principles:
[0078] 1. Data Sharding
[0079] The training dataset is mainly divided based on user needs, such as by simple methods like timestamps or sources, into several subsets called Shards.
[0080] For example, customer service dialogue data can be divided into "Shard A (East China)", "Shard B (North China)", "Shard C (South China)", etc., according to the customer's geographical location.
[0081] Objective: To ensure logical isolation of data, facilitating precise identification of abnormal behavior in the future.
[0082] 2. Sub-model training
[0083] For each subset (Shard), a large language sub-model is trained independently. For example, data from "Shard A" is used only to train the large language sub-model A, data from "Shard B" is used to train sub-model B, and so on.
[0084] The training processes of each language sub-model do not interfere with each other, ensuring a one-to-one correspondence between training data and model output.
[0085] Suppose the training dataset D contains 3 subsets, Shards:
[0086] Shard A: Contains a sample set {x1, x2, ..., x} m};
[0087] Shard B: Contains the sample set {x} m+1 ,x m+2 ,…,x n};
[0088] Shard C: Contains the sample set {x} n+1 ,x n+2 ,…,x p}
[0089] Three large language sub-models, M1, M2, and M3, were trained respectively and are represented as follows:
[0090] M1 = Train(Shard A);
[0091] M2 = Train(Shard B).
[0092] M3 = Train(Shard C).
[0093] For example, the large language sub-model M1 learns to handle the problems of customers in East China based on Shard A, the large language sub-model M2 learns to handle the problems of customers in East China based on Shard B, and the large language sub-model M3 learns to handle the problems of customers in South China based on Shard C.
[0094] 3. Sub-model reasoning and output
[0095] In a preferred embodiment of the present invention, in step S3, as follows: Figure 2 As shown, the calculation of the sensitive word risk confidence score for each candidate answer includes:
[0096] Step S31: Count the total number of times each sensitive word in the preset sensitive word list appears in the candidate answers;
[0097] Step S32: Calculate the proportion of sensitive word characters in the candidate answers based on the total number of occurrences, the text length of each sensitive word appearing in the candidate answers, and the text length of the candidate answers.
[0098] Step S33: Calculate the ratio of the number of each sensitive word appearing in the candidate answer to the text length of the candidate answer, and then perform a weighted summation of the ratio and proportion to obtain the sensitive word risk score;
[0099] Step S34: Adjust the model output confidence score based on the sensitive word risk score to obtain the sensitive word risk confidence score.
[0100] Specifically, let's take the following parameters as an example:
[0101] y i Candidate answers generated by the large language model i;
[0102] S = {s1, s2, ..., s} K}: Sensitive word list, maintained by the security and compliance team;
[0103] Count(s k ,y i ): Sensitive words s k In y i The number of times it appears in;
[0104] Length(·): Text length (can be calculated by the number of characters or words);
[0105] Based on this, the formula for calculating specific gravity is as follows:
[0106]
[0107] Among them, R sens (y i ) represents specific gravity, y i Let S = {s1, s2, ..., s} represent the candidate answers generated by the large language sub-model i. K} represents the list of sensitive words, Length(s) k ) indicates sensitive word s k The length of the text, Count(s) k ,y i ) indicates sensitive word s k In candidate answer y i The number of times L appears in i Indicates candidate answer y i The length of the text.
[0108] This weight indicates the proportion of sensitive word characters in the entire text of the candidate answer.
[0109] Furthermore, the sensitive word risk score R(y) i The calculation formula for ) is as follows:
[0110]
[0111] in, This represents the total number of sensitive words, which is the sum of the number of times each sensitive word appears in the candidate answers generated by the large language sub-model i.
[0112] α, β ≥ 0, and α + β = 1. Generally, α = 0.6 and β = 0.4 are taken as empirical values.
[0113] It is the frequency of sensitive words (the ratio of the number of sensitive words to the text length);
[0114] R sens (y i The above proportion refers to the percentage of sensitive words covered (the percentage of sensitive word characters).
[0115] R(y i The larger the value of ∈ (0,1), the higher the potential sensitivity risk.
[0116] When a user submits a query (e.g., "What is the refund policy for a certain product?"), the request is simultaneously sent to all large language sub-models, generating multiple candidate answers and corresponding model confidence scores. For example, large language sub-model A answers, "Refunds require 7 business days to process," with a confidentiality level (sensitive word risk score) of 0%; large language sub-model B provides a similar answer but with a confidentiality level (sensitive word risk score) of 8%; large language sub-model C's answer differs significantly from the previous two, stating, "Refunds require processing by Liu Bo in the Marketing Department," with a confidentiality level (sensitive word risk score) of 60%. "Liu Bo in the Marketing Department" is considered private information that should not be disclosed.
[0117] 4. Anomaly Detection and Aggregation Analysis
[0118] The outputs of multiple large language sub-models are aggregated, and consistency indicators (such as output differences) and sensitive information are calculated. If significant differences or abnormal fluctuations in confidence levels occur, they are identified as potentially anomalous outputs.
[0119] Specifically, the confidence score of the sensitive word risk for each candidate answer output by the large language sub-model is first calculated. In a preferred embodiment of the present invention, the confidence score of the model output is adjusted based on the sensitive word risk score to obtain the corresponding sensitive word confidence score. The formula for calculating the sensitive word risk confidence score is as follows:
[0120]
[0121] Among them, R(y) i ) represents the sensitive word risk score of the candidate answer corresponding to the large language sub-model i. This represents the confidence score of the model output corresponding to the large language sub-model i, that is, the confidence score of the large language sub-model i for its answer y. i The generated probability confidence level (such as softmax probability) represents the expert's confidence level in each data point; 1 indicates a default "full trust" of the model output, and 0 indicates no trust at all. i This represents the confidence level of the sensitive word risk of the candidate answer corresponding to the large language sub-model i.
[0122] Then, the output consistency index is calculated:
[0123] Define the text similarity function as sim(y i ,y j Given that )∈[0,1] (such as cosine similarity or Jaccard similarity), and M is the total number of large language sub-models, the formula for calculating the average similarity between each candidate answer and the remaining answers is as follows:
[0124]
[0125] If the output of the large language submodel C is found to be significantly different from that of other submodels, then the answer may be abnormal.
[0126] We also need to calculate the overall confidence level mean, i.e., the average confidence level. The calculation formula is as follows:
[0127]
[0128] Furthermore, the formula for calculating the anomaly score is as follows:
[0129]
[0130] Among them, A i Indicates the anomaly score;
[0131] The weights w1, w2, w3 ≥ 0 and w1 + w2 + w3 = 1, which can be adjusted according to needs, such as taking 1 / 3 of each.
[0132] κ≥0 indicates the confidence level fluctuation tolerance factor, with a typical value such as 1.
[0133] τ,s th ∈(0,1), τ represents the average similarity threshold; values below this threshold indicate large output differences, typically 0.5; s th This represents a sensitive risk threshold; exceeding this threshold indicates high risk. A typical value is 0.3.
[0134] c represents the average confidence level; i This represents the confidence level of the sensitive word risk in the candidate answer corresponding to the large language sub-model i; R(y) represents the average similarity; i ) represents the sensitive word risk score of the candidate answer corresponding to the large language sub-model i.
[0135] After calculating the above anomaly scores, the final anomaly determination is performed:
[0136] Set an abnormal threshold A th (e.g., 0.5), if A i >A th If the output of the large language model i is deemed abnormal, it indicates a potential semantic risk or leakage of sensitive information.
[0137] Assumption:
[0138] The candidate answers of large language sub-model A and large language sub-model B have low sensitivity risks, high output similarity, and confidence levels close to the mean.
[0139] The candidate answers of the large language sub-model C contain privacy-sensitive words, resulting in a sensitivity risk R(y). C The value of the candidate answer significantly exceeds the threshold, and the corresponding candidate answer has low text similarity with other large language sub-models, resulting in low overall confidence.
[0140] Based on the above formula for calculating the anomaly score, the anomaly score A of the large language sub-model C is... C It will be significantly greater than the abnormal threshold A th If the corresponding candidate answer is not found, it will be automatically marked as abnormal output, triggering the source tracing and auditing process.
[0141] 5. Tracing the source of the poisoning impact
[0142] Based on the anomaly detection results, the specific subset (one or more) causing the anomaly is identified, which corresponds to the data subset used for training the large language sub-model. Further analysis of the sub-model training logs identifies suspicious training samples.
[0143] For example, through model behavior difference analysis, it was found that samples in Shard C and Shard A contained maliciously constructed combined sentences that induced the model to output incorrect or sensitive information.
[0144] For example, Shard A outputs "Refunds are handled by the Marketing Department" (non-sensitive), Shard B outputs "Refunds require 7 business days to process" (non-sensitive), Shard C outputs "Liu Bo processed 332 orders" (partially sensitive, the department Liu Bo works in is not output here), and Shard D outputs "Refunds require 7 days to process" (non-sensitive).
[0145] The overall language model outputs "Refund will be processed by Liu Bo of the Marketing Department within 7 days" (sensitive).
[0146] 6. Local model repair
[0147] In a preferred embodiment of the present invention, after locking the suspicious training sample in step S5, a local repair operation is further performed, which includes retraining or fine-tuning the corresponding large language sub-model, and then returning to step S2.
[0148] Specifically, for the anomalous subset Shard, only the corresponding large language sub-model is retrained or fine-tuned to avoid retraining the entire model. Anomalous outputs are corrected by adding negative samples or replacing the model.
[0149] If negative samples are augmented and fine-tuned on Shard A and Shard C data, and large language sub-models A and C are retrained, abnormal outputs are significantly reduced without affecting large language sub-models B and D.
[0150] 7. Safety Audit and Accountability Tracing
[0151] In a preferred embodiment of the present invention, the method further includes continuously generating anomaly detection and local repair operation logs during the anomaly detection process of steps S1 to S5, and binding the corresponding suspicious training samples and the corresponding large language sub-models for subsequent security audits and accountability tracing.
[0152] Specifically, it records all anomaly detection, tracing, and remediation operation logs, binds them to specific training data and sub-models, forming a complete audit chain. It supports security auditing and regulatory compliance inquiries.
[0153] For example, if a security incident occurs later, the responsible shard and specific malicious data sample can be quickly traced back to identify the responsible party.
[0154] This invention also provides an exception output management system for large language models based on the SISA architecture, applying the above-mentioned exception output management method, such as... Figure 3 As shown, the exception output management system includes:
[0155] Data sharding and training module 1 is used to divide the training data into multiple subsets and train a large language sub-model for each subset.
[0156] Output generation module 2, connected to data sharding and training module 1, is used to send user questions to various language sub-models simultaneously to obtain candidate answers and corresponding model output confidence scores for each language model.
[0157] Anomaly detection module 3, connected to output generation module 2, is used to calculate the risk confidence of sensitive words for each candidate answer, as well as the text similarity between each candidate answer and the other candidate answers;
[0158] The calculation module 4 is connected to the anomaly detection module 3 and is used to calculate the average confidence of the risk confidence of each sensitive word and the average similarity of each text similarity.
[0159] The source tracing and localization module 5 is connected to the output generation module 2, the anomaly detection module 3 and the calculation module 4 respectively. It is used to calculate the anomaly score corresponding to each candidate answer based on the model output confidence, sensitive word risk confidence, average confidence and average similarity, and to determine the candidate answer with an anomaly score greater than the anomaly threshold as an abnormal output. Then, the subset associated with the large language sub-model corresponding to the candidate answer is locked as a suspicious training sample.
[0160] In a preferred embodiment of the present invention, a local repair module 6 is further included, which is connected to the source tracing and localization module 5, and is used to perform a local repair operation after locking a suspicious training sample. The local repair operation includes retraining or fine-tuning the corresponding large language sub-model.
[0161] In a preferred embodiment of the present invention, an audit and accountability module 7 is further included, which is connected to the source tracing and location module 5 and the local repair module 6 respectively. It is used to continuously generate anomaly detection and local repair operation logs during the anomaly detection process, and bind the corresponding suspicious training samples and the corresponding large language sub-model for subsequent security audit and accountability tracing.
[0162] Specifically, the abnormal output management system for large language models based on the SISA architecture of the present invention achieves closed-loop security management by setting up seven modules that work together in coordination. The modular design ensures the system's scalability and maintainability, supports flexible configuration and upgrades, and adapts to diverse security needs and future technological evolution.
[0163] In summary, this invention introduces the SISA architecture:
[0164] Each data subset corresponds to an independent large language sub-model. These large language sub-models can independently process their respective data subsets during training and inference, and output corresponding results. Furthermore, based on the output results of the large language sub-models, a risk index for each data subset can be further calculated, enabling local data tracking capabilities.
[0165] In the joint inference process of a large language model, the outputs of multiple large language sub-models are aggregated to obtain the final prediction result. Under normal circumstances, the outputs of each large language sub-model should exhibit a certain degree of similarity and stability, and the voting confidence level should remain within a relatively stable range. However, when anomalous data segments exist, the output of the corresponding large language sub-model may become inconsistent with other large language sub-models, and the voting confidence level may also fluctuate abnormally. By comparing the differences in the outputs of each large language sub-model and the fluctuations in voting confidence level, the SISA architecture can accurately locate the data segments causing anomalous behavior, achieving the purpose of impact tracing.
[0166] The SISA architecture processes different subsets of data through multiple large language sub-models. When the outputs of these sub-models are combined under specific conditions, they can reveal potentially sensitive information. For example, suppose there are three seemingly normal text samples, each containing fragments of classified information. Individually, none of the samples reveal any sensitive content. However, when these samples are input into a large language model based on the SISA architecture, the various sub-models process the samples and, through joint inference and semantic analysis, can identify the classified semantics formed by the combination of these samples, thus promptly detecting potential security threats. This achieves the goal of revealing combined anomalies and supports the identification of implicit semantic chains that are "harmless on a single point but classified in combination at multiple points."
[0167] Based on the above functions, the SISA architecture ultimately achieves a fully interpretable path from seemingly normal data input to abnormal semantic output to data fragment backtracking.
[0168] Therefore, the method and system provided by this invention not only solve the problem of generating sensitive information caused by implicit data combination, but also provide data governance means that can be used for security accountability, significantly improving the credibility, controllability and accountability of large language models in security-critical scenarios.
[0169] For example: Suppose an attacker uploads several "normal" text samples containing obscure semantic leaks to Shard C. Individual samples are harmless, and traditional de-identification methods fail to intercept them. After training, the large language sub-model C outputs abnormally sensitive information in multiple joint inferences. Aggregating the outputs of multiple large language sub-models detects inconsistencies between the outputs of large language sub-model C and other large language sub-models, along with abnormal confidence fluctuations, thus identifying an anomaly. Further investigation leads to the relevant malicious samples in the Shard C data. The local remediation module trains on negative samples in Shard C, effectively eliminating the abnormal output. The auditing module records the event, providing a basis for subsequent security accountability.
[0170] The above description is merely a preferred embodiment of the present invention and does not limit the implementation and protection scope of the present invention. Those skilled in the art should realize that any equivalent substitutions and obvious changes made using the content of this specification and illustrations should be included within the protection scope of the present invention.
Claims
1. A method for managing the abnormal output of a large language model based on the SISA architecture, characterized in that, include: Step S1: Divide the training data into multiple subsets, and train a large language sub-model for each subset. Step S2: Send the user's question to each of the large language sub-models simultaneously to obtain the candidate answers and corresponding model output confidence scores of each of the large language models. Step S3: Calculate the sensitive word risk confidence of each candidate answer, and the text similarity between each candidate answer and the remaining candidate answers; Step S4: Calculate the average confidence level of each of the sensitive word risk confidence levels and the average similarity level of each of the text similarities. Step S5: Calculate the anomaly score corresponding to each candidate answer based on the model output confidence, the sensitive word risk confidence, the average confidence and the average similarity, and determine the candidate answer with the anomaly score greater than the anomaly threshold as an abnormal output, and then lock the subset associated with the large language sub-model corresponding to the candidate answer as a suspicious training sample.
2. The abnormal output management method according to claim 1, characterized in that, In step S3, calculating the sensitive word risk confidence level for each candidate answer includes: Step S31: Count the total number of times each sensitive word in the candidate answer appears in the preset sensitive word list; Step S32: Calculate the proportion of sensitive word characters in the candidate answer based on the total number of occurrences, the text length of each sensitive word appearing in the candidate answer, and the text length of the candidate answer. Step S33: Calculate the ratio of the number of each of the sensitive words appearing in the candidate answer to the text length of the candidate answer, and perform a weighted summation of the ratio and the weight to obtain the sensitive word risk score; Step S34: Adjust the model output confidence score based on the sensitive word risk score to obtain the sensitive word risk confidence score.
3. The abnormal output management method according to claim 2, characterized in that, The formula for calculating the specific gravity is as follows: Among them, R sens (y i ) represents the specific gravity, y i Let S = {s1, s2, ..., s} represent the candidate answers generated by the large language sub-model i. K } represents the list of sensitive words, Length(s) k ) indicates sensitive word s k The length of the text, Count(s) k ,y i ) indicates sensitive word s k In candidate answer y i The number of times L appears in i Indicates candidate answer y i The length of the text.
4. The abnormal output management method according to claim 2, characterized in that, The formula for calculating the risk confidence of the sensitive words is as follows: Among them, R(y) i ) represents the sensitive word risk score of the candidate answer corresponding to the large language sub-model i. c represents the confidence level of the model output corresponding to the large language sub-model i. i The risk confidence of the sensitive words in the candidate answer corresponding to the large language sub-model i is represented.
5. The abnormal output management method according to claim 1, characterized in that, The formula for calculating the anomaly score is as follows: Among them, A i The anomaly score represents the score with weights w1, w2, w3 ≥ 0, and w1 + w2 + w3 = 1, κ ≥ 0, and τ, s th ∈(0,1), c represents the average confidence level. i This represents the confidence level of the sensitive word risk of the candidate answer corresponding to the large language sub-model i. R(y) represents the average similarity. i ) represents the sensitive word risk score of the candidate answer corresponding to the large language sub-model i.
6. The abnormal output management method according to claim 1, characterized in that, In step S5, after locking the suspicious training sample, a local repair operation is also performed. The local repair operation includes retraining or fine-tuning the corresponding large language sub-model, and then returning to step S2.
7. The abnormal output management method according to claim 6, characterized in that, It also includes continuously generating anomaly detection and local repair operation logs during the anomaly detection process of steps S1 to S5, and binding the corresponding suspicious training samples and the corresponding large language sub-model for subsequent security audits and accountability tracing.
8. An exception output management system for large language models based on the SISA architecture, characterized in that, The abnormal output management system, which employs the abnormal output management method as described in any one of claims 1-7, comprises: The data sharding and training module is used to divide the training data into multiple subsets and train a large language sub-model for each subset. The output generation module, connected to the data sharding and training module, is used to simultaneously send user questions to each of the large language sub-models to obtain candidate answers and corresponding model output confidence scores for each of the large language models. An anomaly detection module, connected to the output generation module, is used to calculate the sensitive word risk confidence of each candidate answer and the text similarity between each candidate answer and the remaining candidate answers; The calculation module, connected to the anomaly detection module, is used to calculate the average confidence of the risk confidence of each of the sensitive words and the average similarity of each of the text similarities; The source tracing and localization module is connected to the output generation module, the anomaly detection module, and the calculation module, respectively. It is used to calculate the anomaly score corresponding to each candidate answer based on the model output confidence, the sensitive word risk confidence, the average confidence, and the average similarity. The candidate answer with an anomaly score greater than the anomaly threshold is judged as an abnormal output, and then the subset associated with the large language sub-model corresponding to the candidate answer is locked as a suspicious training sample.
9. The abnormal output management system according to claim 8, characterized in that, It also includes a local repair module, which is connected to the source tracing and localization module, and is used to perform a local repair operation after locking the suspicious training sample. The local repair operation includes retraining or fine-tuning the corresponding large language sub-model.
10. The abnormal output management system according to claim 9, characterized in that, It also includes an audit and accountability module, which is connected to the source tracing and localization module and the local repair module, respectively. It is used to continuously generate anomaly detection and local repair operation logs during the anomaly detection process, and bind the corresponding suspicious training samples and the corresponding large language sub-model for subsequent security audit and accountability tracing.