A large language model fine-tuning, security and reasoning integrated system

Abstract

The application relates to a large language model fine-tuning, security and reasoning integrated system, comprising a fine-tuning and security module and an efficient reasoning engine module; the fine-tuning and security module comprises a quadrature LoRA fine-tuning module for solving multi-LoRA adapter merging interference, a red-blue confrontation defense module for solving unknown jailbreak attack defense, and a model counter-learning module for solving unknown jailbreak attack defense and specific data rapid erasing; the efficient reasoning engine module comprises a cascade speculation decoding module for solving reasoning delay and a context-aware quantization module for solving quantization precision collapse. By adopting the application, the comprehensive performance of the model in fine-tuning fusion, security defense, data compliance, reasoning acceleration and quantization deployment and the like can be improved.

Description

Technical Field

[0001] This invention relates to large language model technology, and more particularly to an integrated system for fine-tuning, security, and inference of a large language model. Background Technology

[0002] With the widespread application of Large Language Models (LLMs) based on the Transformer architecture (such as GPT-4 and LLaMA) in the field of natural language processing, the technical challenges of their industrial deployment and practical system applications are becoming increasingly prominent. These challenges cover multiple dimensions, including model fine-tuning, security alignment, data compliance, inference efficiency, and deployment optimization, severely restricting the large-scale implementation and continuous evolution of LLMs. Specifically, existing technologies mainly suffer from the following problems:

[0003] First, in terms of parameter-efficient fine-tuning (PEFT), low-rank adaptation (LoRA) has become the mainstream method. However, in multi-tenant or multi-task scenarios, multiple LoRA adapters trained for different downstream tasks need to be merged into the same base model or dynamically switched. Since the existing LoRA training processes are independent of each other, the weight matrices of different adapters lack orthogonal constraints in the parameter space. Direct superposition or merging can easily cause mutual interference and distortion between weight vectors, i.e., "catastrophic interference," leading to a significant degrade in the performance of each task.

[0004] Secondly, in terms of security defense, current LLM security alignment mechanisms mainly rely on manually labeled reinforcement learning or static red team test datasets. Faced with the endless stream of jailbreak attacks (such as ASCII art injection, role-playing inducement, multi-language encoding attacks, etc.), rule-based or static data-based defense methods show obvious passivity and lag, making it difficult to achieve adaptive and automated adversarial security protection.

[0005] Third, regarding data privacy and copyright compliance, with the implementation of regulations such as the General Data Protection Regulation (GDPR), users' "right to be forgotten" and the demand for the removal of copyrighted text are becoming increasingly urgent. Currently, complete retraining is the only way to ensure that the model's performance is unbiased after the physical removal of data. However, for large language models with trillions of parameters and trillions of tokens in training data, the computational and time costs are extremely high, making it difficult to apply in practice.

[0006] Fourth, regarding inference efficiency, the autoregressive decoding process of LLM generates tokens one by one, making memory bandwidth the main bottleneck. Although traditional speculative decoding introduces a "draft-verification" mechanism, it usually only uses a single small model as the draft model. If the small model is too weak, the draft acceptance rate is low; if it is too strong, the inference latency of the small model itself increases, limiting the overall acceleration effect.

[0007] Fifth, regarding model quantization, quantization operations performed to reduce deployment costs face the risk of accuracy collapse. LLM activation values ​​exhibit significant outliers, with values ​​in specific channels far exceeding the average level. Traditional static quantization (such as uniform W4A8 across all layers) struggles to adapt to such dynamic fluctuations, leading to a severe decline in the perplexity of the quantized model and the performance of downstream tasks.

[0008] In summary, there is an urgent need for a comprehensive optimization solution for the entire lifecycle of large language models that can address the aforementioned technical challenges, thereby improving the overall performance of the models in areas such as fine-tuning and fusion, security defense, data compliance, inference acceleration, and quantization deployment. Summary of the Invention

[0009] The purpose of this invention is to provide an integrated system for fine-tuning, security, and inference of large language models, which addresses the problems of existing models in terms of fine-tuning fusion, security defense, data compliance, inference acceleration, and quantization deployment.

[0010] To address the aforementioned problems, this invention provides an integrated system for fine-tuning, security, and inference of a large language model, the system comprising:

[0011] Fine-tuning and security modules and high-efficiency inference engine modules;

[0012] The fine-tuning and security module includes an orthogonal LoRA fine-tuning module for solving interference from merging multiple LoRA adapters, a red-blue adversarial defense module for defending against unknown jailbreak attacks, and a model anti-learning module for defending against unknown jailbreak attacks and quickly erasing specific data.

[0013] The high-efficiency inference engine module includes a cascaded speculative decoding module to address inference latency and a context-aware quantization module to address quantization accuracy collapse.

[0014] The specific implementation steps of the orthogonal LoRA fine-tuning module include:

[0015] Maintain the adapter repository. When training a new task, initialize a pair of low-rank matrices for the new task, wherein the first low-rank matrix is ​​Gaussian initialized and the second low-rank matrix is ​​zero initialized.

[0016] Singular value decomposition is performed on the low-rank matrix pairs of the trained adapters in the adapter repository to extract the principal singular vectors, and a forbidden subspace matrix is ​​constructed based on the principal singular vectors.

[0017] Construct an orthogonal regularization term to minimize the cosine similarity between the update matrix of the low-rank matrix pair of the new task and the forbidden subspace matrix;

[0018] The orthogonal regularization term is added to the language modeling loss function, and backpropagation is performed through the optimizer to drive the low-rank matrix pair of the new task to find the direction orthogonal to the forbidden subspace matrix when updating the weights. After training is completed, the adapter of the new task is stored in the adapter repository.

[0019] During inference, the low-rank matrix weights of the adapters corresponding to multiple tasks are retrieved from the adapter repository, and the multiple low-rank matrix weights are directly added together. The orthogonality constraint between the low-rank matrix weights makes the cross terms between different tasks approach zero, so as to eliminate interference between tasks.

[0020] The process of constructing the forbidden subspace matrix includes: for each trained adapter, performing singular value decomposition on its first low-rank matrix and second low-rank matrix respectively, and extracting the singular vectors corresponding to the first preset number of singular values ​​of each matrix to form the forbidden subspace matrix.

[0021] The red-blue team adversarial defense module includes: an attacker model configured to generate offensive prompts, a defender model configured to generate responses to input prompts (i.e., the target large language model to be hardened), and an adjudicator model configured to evaluate the offensive prompts and corresponding responses and determine the attack result.

[0022] The red-blue adversarial defense module executes multiple rounds of adversarial training cycles, each round including:

[0023] The attacker model generates a batch of attack prompts based on its current strategy;

[0024] The aggressive prompt is input into the defender model to obtain the corresponding response;

[0025] The referee model evaluates the aggressive prompt and the response pair and outputs a judgment result.

[0026] Based on the determination result, reward signals are calculated for the attacker model and the defender model, respectively.

[0027] Based on the reward signal, the attacker model's strategy is updated using a reinforcement learning algorithm, and the defender model is updated using successful attack samples.

[0028] The specific implementation steps of the model inverse learning module include:

[0029] Receive a sample set of data to be forgotten;

[0030] The gradient of the data to be forgotten with respect to the model parameters is calculated, and the gradient is weighted based on the diagonal approximation of the Fisher information matrix to obtain the natural gradient direction;

[0031] A gradient ascent operation is performed along the natural gradient direction to update the model parameters, and a KL divergence constraint term based on the reference model is introduced into the loss function;

[0032] After the parameters are updated, the data to be forgotten is verified using a member inference attack model. When the probability of the data being identified as a member of the training set approaches a preset threshold, the anti-learning is considered complete.

[0033] The cascaded speculative decoding module includes a draft model, a verification model, and a main model.

[0034] The draft model performs fast inference on the received input sequence to generate a candidate token sequence;

[0035] The candidate token sequence is input into the verification model, which calculates the probability distribution of each position in the sequence and replaces tokens with probabilities below a preset threshold. A corrected candidate token sequence is then generated based on the replaced sequence.

[0036] The corrected candidate token sequence is input into the main model, which performs parallel verification on the corrected candidate token sequence and determines the final output sequence based on the speculative sampling algorithm.

[0037] The system further includes: the key-value cache calculated by the verification model is passed to the main model through a shared memory pool, or the main model only calculates the missing parts of the key-value cache.

[0038] The implementation steps of the context-aware quantization module specifically include:

[0039] The channel statistics of the input activation values ​​of each layer in the large language model are obtained by using calibration data.

[0040] Based on the channel statistics, outlier channels in the input activation values ​​are identified and an outlier channel mask is generated.

[0041] Based on the outlier channel mask, the computation of the linear layer of the large language model is decomposed into a first computation path that processes outlier channels and a second computation path that processes non-outlier channels. The weights and activation values ​​of the first computation path are quantized using a first precision bit width, while the weights and activation values ​​of the second computation path are quantized using a second precision bit width that is lower than the first precision bit width.

[0042] By using a customized single computing kernel, the first computing path and the second computing path are loaded and computed synchronously, and the output results of the two are merged to obtain the final output of the linear layer.

[0043] The channel statistics include the absolute maximum value of activation and kurtosis of each channel; the identification of outlier channels includes: when the product of the absolute maximum value of activation and kurtosis of a channel is greater than a preset threshold, the channel is determined to be an outlier channel, and the corresponding position is marked as valid in the outlier channel mask.

[0044] The present invention has the following beneficial effects:

[0045] First, lossless fusion across multiple tasks. By introducing orthogonal LoRA technology, this invention enables the simultaneous loading of dozens of fine-tuning adapters from different vertical domains onto the same base model. The weight matrices of each adapter maintain an approximately orthogonal relationship in the parameter space, effectively eliminating the catastrophic interference caused by non-orthogonal overlapping weights in traditional methods. Tests show that in multi-task concurrent scenarios, the performance loss between tasks is less than 1%, truly achieving efficient fusion of "one model, multiple functions," and significantly reducing the model maintenance cost for multi-tenant or multi-task deployments.

[0046] Second, proactive immune security. This invention constructs an automated security evolution mechanism based on red-blue team exercises, enabling the model to continuously optimize its robustness through ongoing attack and defense drills. Compared to traditional defense methods that rely on static rules or manual annotation, this mechanism can effectively resist undisclosed "zero-day" jailbreak attacks, significantly improving its generalized defense capabilities against novel and unknown attacks. Experimental data shows that after applying this solution, the model's security compliance rate increases by more than 40%, achieving a leap from passive defense to proactive immunity.

[0047] Third, low cost and compliance. Addressing the needs for data forgetting and copyright removal, this invention employs anti-learning technology. Without retraining the entire model, it precisely erases the memory traces of specific samples in the model through targeted gradient updates or parameter perturbations. Compared to traditional solutions that require weeks or even months of complete retraining, this method reduces the execution time for deleting specific data to minutes. While meeting the timeliness requirements of the "right to be forgotten" under regulations such as the General Data Protection Regulation (GDPR), it significantly reduces computational resource consumption and operating costs.

[0048] Fourth, extreme inference performance. This invention designs a cascaded speculative decoding mechanism, which introduces multi-level draft models (such as from a very small model to a lightweight model) to form a cascaded structure, and filters and verifies the generation path step by step. This scheme overcomes the dilemma of traditional single draft models where it is difficult to balance capability and latency. While ensuring generation quality, it reduces the autoregressive inference latency of large models by 2 to 4 times, significantly improving the real-time response capability of online services and user experience.

[0049] Fifth, high-fidelity compression. This invention employs a context-aware quantization method, which can dynamically sense and adapt to the distribution of outliers in LLM activation values. It adaptively adjusts quantization parameters for different channels or token positions, avoiding the precision collapse caused by outliers in traditional static quantization. While reducing GPU memory usage by more than 50%, the model's perplexity increases by no more than 0.1, and the performance of downstream tasks remains essentially unchanged. This enables edge devices (such as mobile devices and embedded devices) that are originally limited by storage and computing resources to efficiently deploy large language models, significantly expanding their application scenarios.

[0050] In summary, this invention has made groundbreaking progress in multiple dimensions such as multi-task fusion, security defense, data compliance, inference acceleration, and model compression, effectively solving the key technical bottlenecks faced by large language models throughout their entire lifecycle, and possessing significant technological advancement and industrial application value. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of a large language model fine-tuning, security, and inference integrated system according to an embodiment of the present invention;

[0052] Figure 2 This is a flowchart illustrating the implementation steps of the orthogonal LoRA fine-tuning module according to an embodiment of the present invention;

[0053] Figure 3 This is a flowchart of the adversarial training loop in an embodiment of the present invention;

[0054] Figure 4 This is a flowchart illustrating the implementation steps of the model inverse learning module in this embodiment of the invention.

[0055] Figure 5 This is a flowchart of the final output sequence acquisition process in an embodiment of the present invention;

[0056] Figure 6 This is a flowchart illustrating the implementation steps of the context-aware quantization module in this embodiment of the invention. Detailed Implementation

[0057] The present invention will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0058] Figure 1 This is a schematic diagram of a large language model fine-tuning, security, and inference integration system according to an embodiment of the present invention. The system includes:

[0059] Fine-tuning and security module 101 and high-efficiency inference engine module 102;

[0060] The fine-tuning and security module 101 includes an orthogonal LoRA fine-tuning module 1011 for solving interference from merging multiple LoRA adapters, a red-blue adversarial defense module 1012 for defending against unknown jailbreak attacks, and a model anti-learning module 1013 for defending against unknown jailbreak attacks and quickly erasing specific data.

[0061] The high-efficiency inference engine module 102 includes a cascaded speculative decoding module 1021 for solving inference latency and a context-aware quantization module 1022 for solving quantization accuracy collapse.

[0062] From a functional perspective, the five modules—Orthogonal LoRA fine-tuning module, Red-Blue Adversarial Defense module, Model Anti-Learning module, Cascaded Inference Decoding module, and Context-Aware Quantization module—are relatively independent. Each module offers specific solutions to address core pain points of large language models at different stages. For example, the Orthogonal LoRA fine-tuning module resolves weight conflicts in multi-task fine-tuning, the Red-Blue Adversarial Defense module counters jailbreak attacks, the Model Anti-Learning module handles data erasure compliance, the Cascaded Inference Decoding module reduces inference latency, and the Context-Aware Quantization module mitigates accuracy loss during compression. Each module has a clear technical boundary. However, at the macro-architecture level, they are not isolated. Vertically, they form a complete industrial-grade closed loop from training fine-tuning and compliance maintenance to inference deployment. Horizontally, they are categorized into two top-level modules: the "Fine-tuning and Security Module" and the "High-Efficiency Inference Engine Module," corresponding to offline optimization and online services, respectively, constituting a comprehensive and optimized overall solution. In terms of specific deployment locations, the orthogonal LoRA fine-tuning module and the red-blue adversarial defense module mainly act as add-ons in the parameter loading stage and the model interaction layer without changing the original model structure; the model anti-learning module delves into the gradient calculation and optimizer update layer to intervene in the training mechanism; while the cascaded inference decoding module and the context-aware quantization module are embedded in the generation pipeline of autoregressive decoding and the underlying operators of the Transformer linear layer, respectively. In particular, the latter directly achieves fine-grained decomposition and merging of the calculation process through a customized CUDA Kernel, thereby completing a deep adaptation from algorithm to hardware at the engineering level.

[0063] Figure 2This is a flowchart illustrating the implementation steps of the orthogonal LoRA fine-tuning module according to an embodiment of the present invention. The specific implementation steps of the orthogonal LoRA fine-tuning module include:

[0064] S201. Maintain the adapter repository. When training a new task, initialize a pair of low-rank matrices for the new task, wherein the first low-rank matrix is ​​Gaussian initialized and the second low-rank matrix is ​​zero initialized.

[0065] The system persistently maintains an adapter repository, centrally storing all historically trained task adapters. Each adapter consists of a set of low-rank matrices (A... i B i )composition;

[0066] For the new task to be trained, a new set of dedicated low-rank matrices is created, denoted as A. new B new ;

[0067] low-rank matrix A new Gaussian random initialization is used to ensure that the initial parameters have random distribution characteristics;

[0068] low-rank matrix B new Using zero matrix initialization, the weight updates are almost unbiased in the early stages of training;

[0069] The weight increments for all new tasks satisfy the following:

[0070] .

[0071] S202. Perform singular value decomposition on the low-rank matrix pairs of the trained adapters in the adapter repository, extract the principal singular vectors, and construct the forbidden subspace matrix based on the principal singular vectors;

[0072] The process of constructing the forbidden subspace matrix includes: for each trained adapter, performing singular value decomposition on its first low-rank matrix and second low-rank matrix respectively, and extracting the singular vectors corresponding to the first preset number of singular values ​​of each matrix to form the forbidden subspace matrix.

[0073] To avoid the extremely high overhead of orthogonal computation of all parameters, a subspace projection strategy is adopted, and feature extraction is performed only on the low-rank matrix.

[0074] For each historical adapter in the warehouse, the low-rank matrix (A) i B i Perform Singular Value Decomposition (SVD) separately:

[0075] , ;

[0076] After filtering and decomposing, select the top k principal singular values ​​and extract the corresponding principal singular vectors to represent the core feature directions of the historical adapter.

[0077] Integrating the principal singular vectors of all historical adapters, we construct the forbidden subspace matrix U by concatenating them. frob ;

[0078] This forbidden subspace represents the parameter directions that have been occupied by historical tasks and serves as the boundary basis for subsequent orthogonal constraints.

[0079] S203. Construct an orthogonal regularization term to minimize the cosine similarity between the update matrix of the low-rank matrix pair of the new task and the forbidden subspace matrix;

[0080] The overall weight update matrix is ​​as follows: .

[0081] Add a dedicated orthogonal regularization loss term L ortho Core optimization goals:

[0082] Narrowing the new task weight update matrix With the forbidden subspace matrix U frob Cosine similarity;

[0083] The complete loss correlation logic relies on the original formula:

[0084] L total =L LLM +λL ortho ;

[0085] The specific calculation form of orthogonal constraints: L ortho =cos(vec(𝐵 new A new ), U frob );

[0086] λ: Regularization hyperparameter, used to adjust the strength of orthogonal constraints;

[0087] vec: Matrix vectorization operation, converts a two-dimensional matrix into a vector, used for cosine distance calculation;

[0088] The smaller the cosine similarity, the more distant the new task parameter direction is from the historical occupied subspace, and the stronger the orthogonality.

[0089] S204. Add the orthogonal regularization term to the language modeling loss function, and backpropagate through the optimizer to drive the low-rank matrix pair of the new task to find the direction orthogonal to the forbidden subspace matrix when updating the weights. After training is completed, store the adapter of the new task in the adapter repository.

[0090] Modeling loss L using the basic languageLLM To ensure general semantic capabilities, an orthogonal regularization loss L is added. ortho This results in a joint loss:

[0091] L total =L LLM +λcos(vec(𝐵 new A new ), U frob );

[0092] Use the optimizer to backpropagate the total loss and solve for the gradient:

[0093] , ;

[0094] Orthogonal regularized gradients create directional repulsion, constraining... , The iterative update actively avoids the forbidden subspace: the update direction of the new low-rank matrix is ​​forced to fall on the intersection with U. frob An approximately orthogonal blank parameter space;

[0095] To prevent memory overflow, random projection estimation is used during the computation phase, or only the column space of the low-rank matrix is ​​constrained to be orthogonal. After the task converges and training is completed, the final optimized low-rank matrix pair (A) is used. new B new As a brand new adapter, it is stored in the warehouse.

[0096] S205. During inference, the low-rank matrix weights of the adapters corresponding to multiple tasks are retrieved from the adapter repository, and the multiple low-rank matrix weights are directly added together. The orthogonality constraint between the low-rank matrix weights makes the cross terms between different tasks approach zero, so as to eliminate interference between tasks.

[0097] In multi-task joint inference, the weights of the base model and the weights of the multiple adapters are fused by direct summation:

[0098] ;

[0099] 𝑊0 represents the original weights of the pre-trained base model.

[0100] Based on the constraint of orthogonal training throughout the process, the low-rank update matrices of any two different tasks are approximately orthogonal:

[0101] Where, 𝑖 is not equal to 𝑗;

[0102] The cross-interference term in matrix multiplication satisfies:

[0103] ;

[0104] Multiple adapters can be simply stacked to enable multiple tasks. Cross-task interference terms are suppressed to near zero by orthogonal constraints, completely avoiding the conflict problem of merging multiple LoRA adapters.

[0105] The red-blue team adversarial defense module includes: an attacker model configured to generate offensive prompts, a defender model configured to generate responses to input prompts (i.e., the target large language model to be hardened), and an adjudicator model configured to evaluate the offensive prompts and corresponding responses and determine the attack result.

[0106] The red-blue adversarial defense module executes multiple rounds of adversarial training cycles. Figure 3 This is a flowchart of the adversarial training loop in an embodiment of the present invention. Each loop includes:

[0107] S301. The attacker model generates a batch of attack prompts based on its current strategy;

[0108] The attacker model generates a batch of malicious prompts based on the current attack strategy. These prompts may contain complex techniques such as logic traps, role-playing, and code injection, with the goal of inducing the defender model to output harmful content. The generation process requires that the prompts be semantically coherent and not garbled; otherwise, they will be penalized in subsequent reward calculations.

[0109] S302. Input the offensive prompt into the defender model and obtain the corresponding response;

[0110] The offensive prompts generated in the previous step are input into the defender model, i.e., the large language model currently to be hardened. The model generates a corresponding response based on its own training and alignment mechanisms. This response may be a security rejection or guidance, or it may be harmful content that has been successfully induced. The specific result will be determined by the subsequent adjudicator model.

[0111] S303. The referee model evaluates the aggressive prompt and the response pair and outputs a judgment result;

[0112] The referee model (such as GPT-4 or a specialized security classifier) ​​evaluates each pair (attack hint, defender response) to determine whether the attack was successful. If the response contains harmful information, it is considered a "successful attack"; if the model refuses to respond or outputs security guidance, it is considered a "successful defense." This determination provides the basis for subsequent reward calculations.

[0113] S304. Based on the determination result, calculate reward signals for the attacker model and the defender model respectively;

[0114] Based on the judgment results of the referee model, reward signals are calculated for both the attacker and defender models. The attacker's reward is: +1 for a successful attack, -0.5 for a failed attack, and -1 for displaying garbled text. The defender's reward is: +1 for a successful defense, and -1 for a failed defense. These rewards drive the two models to evolve in opposite directions—the attacker aims for successful induction, while the defender aims for successful interception.

[0115] S305. Based on the reward signal, update the attacker model's strategy using a reinforcement learning algorithm, and update the defender model using successful attack samples.

[0116] By utilizing the PPO reinforcement learning algorithm, the attacker's strategy is updated based on reward signals, enabling it to generate more sophisticated and effective attack cues. Simultaneously, all successful attack samples are collected, and the original harmful responses are replaced with secure responses. The defender model is then updated through supervised fine-tuning to patch exposed security vulnerabilities. This step achieves co-evolution between the attacker and the defender.

[0117] Figure 4 The flowchart below shows the implementation steps of the model anti-learning module in this embodiment of the invention. The specific implementation steps of the model anti-learning module include:

[0118] S401. Receive the sample set of data to be forgotten;

[0119] The system receives a forgetting request from an external source and obtains a sample set of data to be forgotten, denoted as D. forget = { (x i , y i )}.

[0120] This sample set is the dataset that the model has learned from during previous training processes, and it is also the core processing object of this anti-learning and data forgetting operation. It provides clear target data basis for subsequent gradient calculation, parameter update and forgetting verification.

[0121] S402. Calculate the gradient of the data to be forgotten with respect to the model parameters, and weight the gradient based on the diagonal approximation of the Fisher information matrix to obtain the natural gradient direction;

[0122] Data sample set D to be forgotten forget Using this as input, the model is inferred forward and solved backward, calculating the gradient of this batch of data with respect to all trainable parameters of the model, denoted as ∇θℒ(D forget The gradient (θ) directly characterizes the degree of influence of the data to be forgotten on the model parameters.

[0123] The Fisher Information Matrix (FIM) is calculated and simplified by using a diagonal approximation, resulting in F ≈ diag(E[∇θlog p(y|x,θ) ∇θlog p(y|x,θ)⊤]), which is used to measure the importance of each model parameter to the original retained knowledge (i.e., the curvature of the parameter space).

[0124] Using the diagonal approximation of the Fisher information matrix, the original gradient calculated above is weighted and corrected to obtain the natural gradient direction ∇̃θ = F⁻¹∇θℒ(D forget , θ).

[0125] This natural gradient direction differs from the ordinary original gradient and can accurately guide parameter updates: it can effectively weaken the model's fitting effect on forgotten data, while maximizing the protection of other normal knowledge already learned by the model and avoiding damage to the model's original core capabilities.

[0126] S403. Perform gradient ascent operation along the natural gradient direction to update the model parameters, and introduce a KL divergence constraint term based on the reference model into the loss function;

[0127] In contrast to the gradient descent (minimizing loss and fitting data) used in conventional model training, this approach performs gradient ascent, updating the model parameters along the natural gradient direction obtained above. The update formula is θ ← θ + η·∇̃θ (η is the learning rate). The purpose is to actively increase the loss corresponding to the data to be forgotten, thereby weakening the model's ability to remember and fit that batch of data.

[0128] To prevent the model from collapsing due to excessive parameter updates, a reference model (i.e., the original model before the forgetting operation) is introduced, and a KL divergence constraint term ℒKL(pref ∥ pθ) is added to the loss function of forgetting optimization.

[0129] The core function of this constraint is to limit the difference in output distribution between the current updated model and the reference model, avoid excessive deviation of model parameters, and ensure that the overall performance and semantic capabilities of the model remain stable while completing data forgetting, without degradation or collapse.

[0130] S405. After the parameters are updated, the data to be forgotten is verified using a member inference attack model. When the probability of the data being identified as a member of the training set approaches a preset threshold, it is determined that anti-learning is complete.

[0131] After updating the model parameters, the forgetting verification process is initiated, employing a Membership Inference Attack (MIA) model to test the data sample set D to be forgotten. forget Conduct specialized tests;

[0132] The core function of a membership inference attack model is to determine whether a single piece of data has ever participated in model training (that is, to distinguish whether the data is a member of the training set);

[0133] Set a preset threshold (the standard threshold is 50%). When the prediction probability that the data to be forgotten is identified as a training set member in the detection result of the MIA model approaches this preset threshold, it indicates that the model can no longer distinguish whether the data has participated in training; at this time, it can be determined that the exclusive memory of the model for the target data to be forgotten has been completely eliminated, and the current unlearning and data forgetting operation is completed.

[0134] Wherein, the cascaded speculative decoding module comprises a draft model, a verification model and a main model;

[0135] Figure 5 is a flow chart of the final output sequence acquisition process in an embodiment of the present invention, the process comprises:

[0136] S501. The draft model performs fast inference on the received input sequence to generate a candidate token sequence;

[0137] The draft model (M1) receives an externally input original sequence and starts a fast inference mode (relying on its distillation training characteristics and video memory resident design to ensure that the inference delay is lower than 5ms);

[0138] The draft model continuously runs for preset steps, such as K=5 steps of inference, to generate a candidate Token sequence with a length of 5, which is recorded as T draft = [t1, t2, t3, t4, t5].

[0139] S502. Input the candidate token sequence into the verification model, the verification model calculates the probability distribution of each position in the sequence, replaces tokens with probability lower than a preset threshold according to the preset threshold, and generates a corrected candidate token sequence based on the replaced sequence;

[0140] Take the candidate sequence T generated in the previous step draft = [t1, t2, t3, t4, t5] as an example, input the complete candidate sequence into the verification model (M2);

[0141] The verification model (M2) calculates the conditional probability distribution P(t i | t<i) for Token ti (i=1,2,3,4,5) at each position in the sequence, wherein t<i represents all prefix Tokens before the i-th Token;

[0142] Set a preset probability threshold γ1, and judge and screen the conditional probability of each Token:

[0143] If P M2 (t i | t<i) > γ1 (the probability is higher than the threshold), retain the Token t at this position i without modification;

[0144] If P M2 (t i | t<i) < γ1 (the probability is lower than the threshold), the verification model (M2) replaces this Token t i to generate a reasonable corrected Token t' i ;

[0145] After replacement, the verification model (M2) continues to generate subsequent additional tokens based on the corrected prefix sequence (including the replaced Token), and finally forms a corrected candidate Token sequence, denoted as T' draft ;

[0146] Example: If t3 is corrected to t'3, and t'4 and t'5 are additionally generated subsequently, then T' draft = [t1, t2, t'3, t'4,t'5], in this case the verification model also plays the role of advanced draft generation and improves sequence quality.

[0147] S503. Input the corrected candidate token sequence into the main model, and the main model performs parallel verification on the corrected candidate token sequence, and determines the final output sequence based on the speculative sampling algorithm.

[0148] Input the corrected candidate Token sequence T' generated in the previous step draft into the main model (M3), and the main model starts the parallel verification mode;

[0149] The main model only performs one forward propagation to calculate T' in parallel draft the probability distributions of all Token positions therein, without step-by-step calculation Token by Token, which improves verification efficiency;

[0150] Adopt the speculative sampling algorithm to judge and determine the finally accepted sequence length L based on the probability distribution calculated by the main model;

[0151] Since T' draftAfter pre-correction by the validated model (M2), the sequence quality has been significantly improved. Therefore, the main model (M3) can achieve an acceptance rate of over 80% for this sequence, ultimately outputting a high-quality sequence of length L, completing the cascade generation process. To avoid redundant computation and improve efficiency, the system is designed with a shared memory pool. If the KV cache computed by M2 passes validation, it can be directly passed to M3 (when dimension compatibility is maintained), or M3 can compute only the missed parts. This design supports efficient execution of the steps.

[0152] Figure 6 The flowchart below shows the implementation steps of the context-aware quantization module in this embodiment of the invention. The implementation steps of the context-aware quantization module specifically include:

[0153] S601. Obtain the channel statistics of the input activation values ​​of each layer in the large language model through calibration data;

[0154] Before inference begins, a small amount of calibration data (i.e., calibration set) is selected and input into the large language model to allow the model to run an inference process once.

[0155] For each Transformer layer in the model, extract the input activation value tensor of that layer, denoted as X (the dimension adapts to the model layer and represents the feature information of the input of that layer).

[0156] Channel-level statistics are performed on the input activation value tensor X, and two core channel statistics are calculated:

[0157] The absolute maximum value of each channel is denoted as |X|. max This is used to measure the overall strength of the channel's activation value;

[0158] The kurtosis of each channel, denoted as K, is used to measure the steepness of the distribution of activation values ​​in that channel. The greater the deviation of the kurtosis from the normal range, the more abnormal the data of that channel is.

[0159] Record the |X| of all channels max Together with kurtosis, they form a complete set of channel statistics, providing a criterion for subsequent outlier channel identification.

[0160] S602. Identify outlier channels in the input activation value based on the channel statistics and generate an outlier channel mask;

[0161] A pre-defined outlier detection threshold τ is set (the threshold can be adaptively adjusted according to the model type and the distribution of calibration data).

[0162] Iterate through each channel j and calculate the kurtosis K of that channel. j The comparison is made with the threshold τ, and the judgment criterion is: if K of a certain channel j jIf K > τ, then the activation value distribution of this channel is abnormal and it is marked as an "outlier channel"; if K j If ≤ τ, then it is a normal channel (not an outlier channel);

[0163] Construct a sparse outlier mask vector M with dimension D (D equals the total number of channels), i.e., M ∈ {0, 1}. D The assignment rule for the mask vector is: M(j)=1 represents that channel j is an outlier channel, and M(j)=0 represents that channel j is a normal channel;

[0164] In practical applications, the proportion of outlier channels is usually less than 1%, so the mask vector is sparsely distributed. Subsequently, the outlier channels and normal channels can be quickly distinguished by the mask, so as to achieve accurate splitting of the calculation path.

[0165] S603. Based on the outlier channel mask, the computation of the linear layer of the large language model is decomposed into a first computation path for processing outlier channels and a second computation path for processing non-outlier channels, wherein the weights and activation values ​​of the first computation path adopt a first precision bit width, and the weights and activation values ​​of the second computation path are quantized using a second precision bit width lower than the first precision bit width.

[0166] The core calculation formula for the linear layer in a large language model is Y = X·W, where X is the input activation value tensor, W is the linear layer weight matrix, and Y is the linear layer output.

[0167] Based on the outlier channel mask M, both the input activation value X and the weight matrix W are split into two parts, corresponding to two computation paths. After the split, the linear layer calculation formula becomes: Y = X·W = (X outlier ·W outlier ) + (X normal ·W normal );

[0168] Among them, X outlier W is the activation value subtensor corresponding to the outlier channel. outlier X is the weight submatrix corresponding to the outlier channel; normal W is the activation value subtensor corresponding to the normal channel. normal This is the weight submatrix corresponding to the normal channel;

[0169] First computational path (outlier channel processing): X outlier and W outlier The first precision bit width (high precision) is used, usually in FP16 or INT8 format; although the path has high precision, the outlier channel accounts for a very low percentage (<1%), and the corresponding computational load is very small, so it will not increase the computational burden too much.

[0170] Second computation path (non-outlier channel processing): X normal and W normal The second precision bit width (low precision) is used, and quantization is usually performed in INT4 format. Since outliers have been removed from the normal channels, the distribution of activation values ​​and weights is very compact. The quantization noise brought by INT4 quantization is extremely low and will not affect the model inference accuracy.

[0171] The dual-path splitting design reduces most of the computational load through low-precision quantization (normal channels account for >99%), while protecting outlier channels through high-precision paths to avoid precision loss.

[0172] S604. Using a customized single computing kernel, the first computing path and the second computing path are loaded and computed synchronously, and the output results of the two are merged to obtain the final output of the linear layer.

[0173] Write a custom CUDA Kernel (runtime kernel). The core advantage of this kernel is that it can load high-precision data (FP16 / INT8) of the first computation path and low-precision data (INT4) of the second computation path in a single kernel launch, without having to start the kernel twice, thus greatly improving computational efficiency.

[0174] Kernel internal execution logic (relying on GPU hardware resources):

[0175] (1) Utilize the GPU's Tensor Core to process the INT4 matrix multiplication (X) in the second computation path. normal ·W normal Tensor Core excels at low-precision matrix operations and can quickly complete most computational tasks;

[0176] (2) Utilize the GPU's CUDA Cores to process the FP16 / INT8 sparse vector multiplication (X) in the first computation path. outlier ·W outlier It adapts to the sparse characteristics of outlier channels and efficiently completes a small number of high-precision calculations.

[0177] After the calculations for both paths are completed, the calculation results (X) of the two paths are entered into the GPU's accumulator. outlier ·W outlier and X normal ·W normal The X and X are merged to obtain the final output Y of the linear layer, i.e., Y = (X... outlier ·W outlier )+ (X normal ·W normal ), to complete the mixing accuracy calculation of the linear layer.

[0178] In the original implementation steps, "dynamic bit width adjustment" is an optional optimization step. The core is to use a lightweight predictor to predict whether the current layer should enable high-precision protection based on the output of the previous layer: if the token activation value distribution is stable, the entire layer is switched to INT4 for further acceleration without affecting the core process of S601–S604.

[0179] The present invention has the following beneficial effects:

[0180] First, lossless fusion across multiple tasks. By introducing orthogonal LoRA technology, this invention enables the simultaneous loading of dozens of fine-tuning adapters from different vertical domains onto the same base model. The weight matrices of each adapter maintain an approximately orthogonal relationship in the parameter space, effectively eliminating the catastrophic interference caused by non-orthogonal overlapping weights in traditional methods. Tests show that in multi-task concurrent scenarios, the performance loss between tasks is less than 1%, truly achieving efficient fusion of "one model, multiple functions," and significantly reducing the model maintenance cost for multi-tenant or multi-task deployments.

[0181] Second, proactive immune security. This invention constructs an automated security evolution mechanism based on red-blue team exercises, enabling the model to continuously optimize its robustness through ongoing attack and defense drills. Compared to traditional defense methods that rely on static rules or manual annotation, this mechanism can effectively resist undisclosed "zero-day" jailbreak attacks, significantly improving its generalized defense capabilities against novel and unknown attacks. Experimental data shows that after applying this solution, the model's security compliance rate increases by more than 40%, achieving a leap from passive defense to proactive immunity.

[0182] Third, low cost and compliance. Addressing the needs for data forgetting and copyright removal, this invention employs anti-learning technology. Without retraining the entire model, it precisely erases the memory traces of specific samples in the model through targeted gradient updates or parameter perturbations. Compared to traditional solutions that require weeks or even months of complete retraining, this method reduces the execution time for deleting specific data to minutes. While meeting the timeliness requirements of the "right to be forgotten" under regulations such as the General Data Protection Regulation (GDPR), it significantly reduces computational resource consumption and operating costs.

[0183] Fourth, extreme inference performance. This invention designs a cascaded speculative decoding mechanism, which introduces multi-level draft models (such as from a very small model to a lightweight model) to form a cascaded structure, and filters and verifies the generation path step by step. This scheme overcomes the dilemma of traditional single draft models where it is difficult to balance capability and latency. While ensuring generation quality, it reduces the autoregressive inference latency of large models by 2 to 4 times, significantly improving the real-time response capability of online services and user experience.

[0184] Fifth, high-fidelity compression. This invention employs a context-aware quantization method, which can dynamically sense and adapt to the distribution of outliers in LLM activation values. It adaptively adjusts quantization parameters for different channels or token positions, avoiding the precision collapse caused by outliers in traditional static quantization. While reducing GPU memory usage by more than 50%, the model's perplexity increases by no more than 0.1, and the performance of downstream tasks remains essentially unchanged. This enables edge devices (such as mobile devices and embedded devices) that are originally limited by storage and computing resources to efficiently deploy large language models, significantly expanding their application scenarios.

[0185] In summary, this invention has made groundbreaking progress in multiple dimensions such as multi-task fusion, security defense, data compliance, inference acceleration, and model compression, effectively solving the key technical bottlenecks faced by large language models throughout their entire lifecycle, and possessing significant technological advancement and industrial application value.

[0186] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A system integrating fine-tuning, security, and inference for a large language model, characterized in that, include: Fine-tuning and security modules and high-efficiency inference engine modules; The fine-tuning and security module includes an orthogonal LoRA fine-tuning module for solving interference from merging multiple LoRA adapters, a red-blue adversarial defense module for defending against unknown jailbreak attacks, and a model anti-learning module for defending against unknown jailbreak attacks and quickly erasing specific data. The high-efficiency inference engine module includes a cascaded speculative decoding module to address inference latency and a context-aware quantization module to address quantization accuracy collapse.

2. The integrated system for fine-tuning, security, and inference of a large language model according to claim 1, characterized in that, The specific implementation steps of the orthogonal LoRA fine-tuning module include: Maintain the adapter repository. When training a new task, initialize a pair of low-rank matrices for the new task, wherein the first low-rank matrix is ​​Gaussian initialized and the second low-rank matrix is ​​zero initialized. Singular value decomposition is performed on the low-rank matrix pairs of the trained adapters in the adapter repository to extract the principal singular vectors, and a forbidden subspace matrix is ​​constructed based on the principal singular vectors. Construct an orthogonal regularization term to minimize the cosine similarity between the update matrix of the low-rank matrix pair of the new task and the forbidden subspace matrix; The orthogonal regularization term is added to the language modeling loss function, and backpropagation is performed through the optimizer to drive the low-rank matrix pair of the new task to find the direction orthogonal to the forbidden subspace matrix when updating the weights. After training is completed, the adapter of the new task is stored in the adapter repository. During inference, the low-rank matrix weights of the adapters corresponding to multiple tasks are retrieved from the adapter repository, and the multiple low-rank matrix weights are directly added together. The orthogonality constraint between the low-rank matrix weights makes the cross terms between different tasks approach zero, so as to eliminate interference between tasks.

3. The integrated system for fine-tuning, security, and inference of a large language model according to claim 2, characterized in that, The process of constructing the forbidden subspace matrix includes: for each trained adapter, performing singular value decomposition on its first low-rank matrix and second low-rank matrix respectively, and extracting the singular vectors corresponding to the first preset number of singular values ​​of each matrix to form the forbidden subspace matrix.

4. The integrated system for fine-tuning, security, and inference of a large language model according to claim 1, characterized in that, The red-blue team adversarial defense module includes: an attacker model configured to generate offensive prompts, a defender model configured to generate responses to input prompts (i.e., the target large language model to be security hardened), and an adjudicator model configured to evaluate the offensive prompts and corresponding responses and determine the attack result.

5. The integrated system for fine-tuning, security, and inference of a large language model according to claim 4, characterized in that, The red-blue adversarial defense module executes multiple rounds of adversarial training cycles, where each round includes: The attacker model generates a batch of attack prompts based on its current strategy; The aggressive prompt is input into the defender model to obtain the corresponding response; The referee model evaluates the aggressive prompt and the response pair and outputs a judgment result. Based on the determination result, reward signals are calculated for the attacker model and the defender model, respectively. Based on the reward signal, the attacker model's strategy is updated using a reinforcement learning algorithm, and the defender model is updated using successful attack samples.

6. The integrated system for fine-tuning, security, and inference of a large language model according to claim 1, characterized in that, The specific implementation steps of the model anti-learning module include: Receive a sample set of data to be forgotten; The gradient of the data to be forgotten with respect to the model parameters is calculated, and the gradient is weighted based on the diagonal approximation of the Fisher information matrix to obtain the natural gradient direction; A gradient ascent operation is performed along the natural gradient direction to update the model parameters, and a KL divergence constraint term based on the reference model is introduced into the loss function; After the parameters are updated, the data to be forgotten is verified using a member inference attack model. When the probability of the data being identified as a member of the training set approaches a preset threshold, the anti-learning is considered complete.

7. The integrated system for fine-tuning, security, and inference of a large language model according to claim 1, characterized in that, The cascaded speculative decoding module includes a draft model, a verification model, and a main model; The draft model performs fast inference on the received input sequence to generate a candidate token sequence; The candidate token sequence is input into the verification model, which calculates the probability distribution of each position in the sequence and replaces tokens with probabilities below a preset threshold. A corrected candidate token sequence is then generated based on the replaced sequence. The corrected candidate token sequence is input into the main model, which performs parallel verification on the corrected candidate token sequence and determines the final output sequence based on the speculative sampling algorithm.

8. The integrated system for fine-tuning, security, and inference of a large language model according to claim 7, characterized in that, The system further includes: the key-value cache calculated by the verification model is passed to the main model through a shared memory pool, or the main model only calculates the parts of the key-value cache that are not hit.

9. The integrated system for fine-tuning, security, and inference of a large language model according to claim 1, characterized in that, The specific implementation steps of the context-aware quantization module include: The channel statistics of the input activation values ​​of each layer in the large language model are obtained by using calibration data. Based on the channel statistics, outlier channels in the input activation values ​​are identified and an outlier channel mask is generated. Based on the outlier channel mask, the computation of the linear layer of the large language model is decomposed into a first computation path that processes outlier channels and a second computation path that processes non-outlier channels. The weights and activation values ​​of the first computation path are quantized using a first precision bit width, while the weights and activation values ​​of the second computation path are quantized using a second precision bit width that is lower than the first precision bit width. By using a customized single computing kernel, the first computing path and the second computing path are loaded and computed synchronously, and the output results of the two are merged to obtain the final output of the linear layer.

10. The integrated system for fine-tuning, security, and inference of a large language model according to claim 9, characterized in that, The channel statistics include the absolute maximum value of activation and kurtosis of each channel; the identification of outlier channels includes: when the product of the absolute maximum value of activation and kurtosis of a channel is greater than a preset threshold, the channel is determined to be an outlier channel, and the corresponding position is marked as valid in the outlier channel mask.

Images