A privacy-forgetting method for large-scale vertical domain models guided by dual Fisher matrices, and its application in terminals.

Abstract

This invention relates to the fields of artificial intelligence and privacy protection technology, and discloses a privacy-preserving forgetting method and terminal for large-scale vertical domain models guided by dual Fisher matrices. This method separates two subsets of text sequence data from the original training corpus of a large-scale vertical domain language model: a retained dataset and a forgotten dataset. The retained dataset contains vertical domain-specific data without privacy information, while the forgotten dataset contains privacy-sensitive data to be eliminated. Based on information geometry theory, the retained Fisher information matrix and the forgotten Fisher information matrix of the model parameters are calculated on the retained dataset and the forgotten dataset, respectively. Based on the synergistic analysis of the retained and forgotten Fisher information matrices, a parameter forgetting guidance function is constructed, and the guidance function values ​​of all model parameters are calculated, thereby constructing a standardized guidance vector. This invention achieves a refined balance between privacy elimination and capability preservation in large-scale language model privacy forgetting.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and privacy protection technology, specifically to a method and terminal for privacy forgetting of large vertical domain models guided by dual Fisher matrices. Background Technology

[0002] With the deep penetration of artificial intelligence technology, large-scale language models have become the core technological foundation for key vertical fields such as intelligent medical diagnosis, intelligent legal consultation, financial risk prediction, and public opinion analysis. These domain-specific large language models, through continuous pre-training and supervised fine-tuning on industry corpora ranging from billions to hundreds of billions of tokens, have gained a deep understanding of professional knowledge and complex reasoning capabilities. They can provide auxiliary diagnostic suggestions for clinicians, generate legal opinions for lawyers, and predict market trends for financial analysts. However, to achieve practical performance levels, these large-scale models in vertical fields must ingest massive amounts of real-world data during the training phase, including data from medical electronic medical records systems, court judgments, social media user posting records, and internal enterprise operational documents. These training corpora inevitably contain a large amount of personal privacy information, such as patients' genetic testing reports and medication records, the identity information and family relationships of litigants, and the geographical location and workplace of social media users. Due to the inherent memory effect of deep neural networks, this privacy information is encoded in a distributed manner in the model's tensors with hundreds of millions of parameters, creating a deep-seated risk of privacy leakage. During the model inference phase, when users conduct exploratory queries using carefully designed prompts, the model may perform cross-text association and semantic splicing of fragmented privacy information memorized during training, outputting highly sensitive privacy data combinations including real names, ID numbers, medical record details, and social relationship networks. This could violate relevant laws and regulations regarding data minimization and the "right to be forgotten," posing a potential risk to personal privacy and data security.

[0003] To address the issue of privacy data removal in large language models, academia and industry have proposed the Machine Unlearning paradigm. This aims to allow trained models to selectively "forget" the influence of specific training samples, as if these samples had never been used in training. Current mainstream model forgetting methods include full retraining, gradient ascent forgetting, and influence function approximation. Full retraining physically removes privacy data from the training corpus and then trains the model from scratch. While this completely eliminates privacy impact, for large vertical domain models with hundreds of billions of parameters, a single retraining session requires thousands of GPU days (the computational power of one GPU for one day), resulting in a time cost of several weeks and economic costs of millions of yuan. Furthermore, this process must be repeated every time a new privacy removal request is received, limiting its practical application in engineering. Gradient ascent forgetting methods actively increase the model's predictive loss to achieve a forgetting effect by performing gradient ascent optimization on the data to be forgotten. However, this method lacks fine-grained differentiation of parameter importance, causing indiscriminate perturbations to the model's parameter space while eliminating privacy memories. This leads to severe performance degradation in core vertical tasks such as medical diagnosis and legal text retrieval, with accuracy drops of 20% to 40%, resulting in knowledge forgetting problems. Influence function methods attempt to locate parameters that need correction by calculating the impact of training samples on model parameters. However, this method requires calculating the inverse of the Hessian matrix, which has a computational complexity of O(n³) for large models with billions of parameters, facing memory overflow and computational infeasibility issues in practical applications. More importantly, existing methods generally lack verifiable mechanisms to guarantee the forgetting effect, failing to prove that privacy data has been completely forgotten rather than merely superficially suppressed, posing a security risk that privacy information can still be extracted through adversarial hint attacks. Summary of the Invention

[0004] To address the technical problem that existing large language model privacy forgetting methods struggle to achieve a fine balance between privacy elimination and capability preservation, thus limiting the application of these models in certain vertical domains, this invention provides a vertical domain large model privacy forgetting method and terminal based on dual Fisher matrix guidance.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] This invention discloses a privacy forgetting method for large vertical domain models guided by a dual Fisher matrix, comprising the following steps, namely S1 to S5.

[0007] S1. From the original training corpus of the vertical domain large language model, two subsets of text sequence data are separated, namely the retained dataset and the forgotten dataset; wherein, the retained dataset contains vertical domain professional data without privacy information, and the forgotten dataset contains privacy-sensitive data to be eliminated.

[0008] S2. Based on information geometry theory, calculate the retained Fisher information matrix and the forgotten Fisher information matrix of the model parameters on the retained dataset and the forgotten dataset, respectively.

[0009] S3. Based on the synergistic analysis of the retained Fisher information matrix and the forgotten Fisher information matrix, construct a parameter forgetting guidance function, calculate the guidance function values ​​of all parameters of the model, and construct a standardized guidance vector accordingly; wherein, the guidance function value of each parameter is used to characterize the forgetting benefit-retention cost ratio of that parameter, the larger the value, the higher the marginal benefit of forgetting that parameter for privacy elimination, and the lower the marginal loss to domain performance.

[0010] S4. Based on the statistical distribution characteristics of the standardized guiding vector, the parameter space is adaptively divided into multiple subspaces with different forgetting priorities.

[0011] S5. Design differentiated parameter-selective forgetting strategies for the multiple subspaces respectively, and achieve the forgetting of privacy data and the preservation of domain knowledge by jointly optimizing and updating parameters.

[0012] As a further improvement to the above scheme, in step S2, for model number... Parameters It retains Fisher information as Forgot Fisher information The formulas are defined as follows:

[0013] ;

[0014] ;

[0015] In the formula, , This represents the total number of parameters in the model. Indicates that in preserving the dataset Mathematical expectation on To preserve the dataset; In the forgotten dataset Mathematical expectation on Forgotten dataset; For partial derivative operators; For the model in the parameter set Based on the input Predicted output The conditional probability distribution, , Given a sequence of text as input, Output the target sequence.

[0016] As a further improvement to the above scheme, in step S2, Monte Carlo estimation is used to sample several batches of samples from the retained dataset and the forgotten dataset respectively. For each sampled sample, forward propagation and backward propagation of the model are performed to calculate the gradient value of the model loss function with respect to each model parameter. The squared gradient value of each parameter is averaged over all sampled samples, and the average value is the approximate value of the Fisher information value of that parameter, which is used for subsequent calculations.

[0017] The strategy employed is a layer-by-layer accumulation approach: the large language model for the vertical domain is grouped according to Transformer layers, and the model has a total of Layer, number The parameter set of a layer is denoted as The total Fisher information matrix is ​​then a concatenation of the matrices from each layer, expressed as follows:

[0018] ;

[0019] ;

[0020] In the formula, The reserved Fisher information matrix; The forgotten Fisher information matrix; and The first The retained Fisher information matrix and the forgotten Fisher information matrix of the layer Among them, the first The Fisher information matrix of a layer is a vector composed of the Fisher information values ​​of all parameters of that layer, using a diagonal approximation method.

[0021] After the Fisher information submatrix of each layer is calculated, the parameters of that layer are loaded from the host memory to the graphics processor's video memory. Gradient calculation is performed on the graphics processor to obtain the Fisher information submatrix of that layer. Immediately afterward, the Fisher information submatrix of that layer is written to the persistent storage medium through the file system interface and the graphics processor's video memory is released for use by the next layer. After all layers are calculated, the submatrixes of each layer are read from the persistent storage medium and concatenated into a complete Fisher information matrix.

[0022] As a further improvement to the above scheme, in step S3, the expression of the parameter forgetting guidance function is:

[0023] ;

[0024] In the formula, For parameters The value of the guiding function; and It is a non-linear adjustment index; This is a numerical stability smoothing term; where the numerator term... The privacy risk level used to quantify parameters, index Differences in importance used to amplify or suppress forgetting; denominator term The domain knowledge contribution used to quantify parameters serves as a forgetting constraint, and the exponent... Control the intensity of penalties for maintaining importance.

[0025] As a further improvement to the above scheme, the process of constructing the standardized guiding vector in step S3 is as follows:

[0026] Define the standardized bootstrap function:

[0027] ;

[0028] In the formula, for The result after standardization; The mean of the logarithmic indices; The standard deviation is denoted as .

[0029] The standardized guiding function is used to standardize all guiding function values ​​to obtain the standardized guiding vector. .

[0030] As a further improvement to the above scheme, step S4 includes the following specific steps, namely S41 to S43.

[0031] S41. Calculate the 75th percentile of the guiding function value based on the standardized guiding vector. and the 25th percentile .

[0032] S42. Set adjustable partition boundary coefficient and The partition threshold is calculated using the following formula:

[0033] ;

[0034] ;

[0035] In the formula, This is used to control the aggressiveness of partitioning; For high partition threshold, This is the low partition threshold.

[0036] S43. Based on the partitioning threshold, the parameter space is divided into three mutually exclusive subspaces: a high bootstrap value region, a medium bootstrap value region, and a low bootstrap value region, as expressed below:

[0037] ;

[0038] ;

[0039] ;

[0040] In the formula, Represents the parameter space. For parameters, Indicates parameters The result of standardizing the value of the guiding function; This is the high-leading-value region, containing parameters that are highly sensitive to privacy data but contribute little to domain tasks; This is the guiding value region, containing parameters that simultaneously encode privacy information and expertise; This is the low-leading-value region, containing parameters that are important for domain tasks but weakly correlated with privacy data.

[0041] As a further improvement to the above scheme, in step S43, after the partitioning is completed, the number of parameters in each region is also counted. , , The respective proportions are determined to ensure that the partitioning results meet the expected distribution.

[0042] As a further improvement to the above scheme, in step S5, for the high guiding value region... Define the strong forgetting loss function:

[0043] ;

[0044] In the formula, For strong forgetting loss; Represents the forgotten dataset The number of samples; For parameters in the high-leading value region, These are the initial parameter values; The regularization coefficient is used. Indicates input-output sample pairs. Parameters for high-guided-value regions The conditional probability distribution of the model is as follows. Given a sequence of text as input, Output the sequence for the target; It is an L2 norm.

[0045] For the middle guiding value area Define the balanced loss function:

[0046] ;

[0047] In the formula, To balance the losses; For cross-entropy loss, and Let be the weighting coefficient, satisfying ; The parameters are in the guided value area; Indicates the retention of datasets The number of samples; For the guiding value region parameter The conditional probability distribution of the model.

[0048] For low guiding value areas Define the protection loss function:

[0049] ;

[0050] In the formula, To protect against losses; The output distribution of a large language model in a vertical domain before forgetting; Parameters for low-guide value regions The conditional probability distribution of the model; Let KL divergence be a metric. This is the standard mathematical notation for the KL divergence.

[0051] As a further improvement to the above scheme, in step S5, a globally unified selective forgetting optimization objective is constructed:

[0052] ;

[0053] In the formula, and This represents the regional weighting coefficient.

[0054] During the joint optimization process, after each preset number of iterations, the retained Fisher information matrix and the forgotten Fisher information matrix are recalculated based on the current model state. The parameter forgetting guidance function and parameter space partitioning are updated according to the recalculated Fisher information matrix until convergence.

[0055] The present invention also discloses a computer terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the steps of the vertical domain large model privacy forgetting method guided by dual Fisher matrices as described above.

[0056] Compared with the prior art, the beneficial effects of the present invention are:

[0057] 1. The proposed dual Fisher matrix guidance method enables two-dimensional quantitative analysis of the parameter space of a large language model. By calculating Fisher information matrices on both the retained and forgotten datasets, this invention can simultaneously characterize the functional attributes of each parameter: the retained Fisher information matrix reflects the parameter's contribution to the vertical domain task, while the forgotten Fisher information matrix reflects the parameter's encoding strength for privacy data. A parameter forgetting guidance function constructed based on these two matrices assigns a quantifiable forgetting priority to each parameter, representing the ratio of forgetting gains to retention costs, thus solving the problem of singular parameter importance assessment in existing technologies. Implementation examples demonstrate that this method can accurately identify parameter subspaces that are highly sensitive to privacy data but contribute little to the domain task. The targeted forgetting of these parameters makes privacy data difficult to identify by member inference attacks, while maintaining the model's performance on the vertical domain task, outperforming traditional gradient ascent methods.

[0058] 2. The parameter three-part adaptive forgetting algorithm designed in this invention effectively resolves the contradiction between forgetting intensity and performance preservation. Based on the statistical distribution characteristics of the standardized guiding vector, the system automatically divides the parameter space into three subspaces: a high guiding value region, a medium guiding value region, and a low guiding value region. A differentiated loss function is configured for each subspace: the high guiding value region uses a strong forgetting loss to achieve gradient ascent forgetting; the medium guiding value region uses a balanced loss to weightedly combine forgetting and retention objectives; and the low guiding value region uses a protective loss to lock the output distribution through KL divergence. This partitioning strategy enables the algorithm to accurately update parameters related to privacy data while protecting key parameters carrying domain knowledge. Furthermore, the dynamic update mechanism introduced in this invention recalculates the double Fisher matrix and updates the parameter partitions every preset number of iterations, ensuring that the forgetting strategy is always based on the latest state of the model. The parameter partition ratio can be flexibly adjusted according to specific business needs.

[0059] 3. The privacy-forgetting method implemented in this invention can solve the timeliness technical challenge of large-scale models in vertical domains regarding privacy compliance. For privacy-sensitive vertical business scenarios such as medical diagnosis and legal consultation, large language models may encode medical records of tens of thousands of patients or thousands of confidential legal documents during training. When individual data subjects exercise their deletion rights, traditional full retraining methods require retraining the entire model from scratch. Each retraining cycle is long, and this process must be repeated for each new deletion request, leading to a backlog of privacy deletion requests in actual projects that cannot be processed in a timely manner. The parameterized forgetting method of this invention, by accurately locating and updating the model parameter subspace, shortens the execution time of a single forgetting operation to the hour level and supports batch processing of multiple deletion requests, enabling timely responses to data subjects' rights claims. This technological breakthrough provides compliance assurance for the deployment of large language models in strictly regulated vertical industries such as healthcare, finance, and law, resolving the conflicting technical challenges between privacy protection regulations and AI technology applications, and transforming the application of large models in highly privacy-sensitive fields from compliance-infeasible to technically feasible. Attached Figure Description

[0060] Figure 1 This is a flowchart of the privacy forgetting method for large vertical domain models guided by dual Fisher matrices in Embodiment 1 of the present invention.

[0061] Figure 2 This is an example diagram of the selective privacy forgetting process in the large medical vertical domain model in Embodiment 1 of the present invention.

[0062] Figure 3 This is a schematic diagram of the structure of the computer terminal in Embodiment 2 of the present invention. Detailed Implementation

[0063] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] Example 1

[0065] Existing privacy-preserving forgetting techniques for large language models have the following limitations. First, there is the problem of single-dimensional parameter importance assessment. Existing methods, such as continuous learning techniques based on elastic weight consolidation, primarily focus on protecting parameters important to the original task. However, model forgetting scenarios need to address a dual constraint: identifying both parameters encoding privacy information and parameters carrying domain knowledge. Since some parameters may possess dual functional attributes, single-dimensional importance assessment struggles to accurately define the boundaries of forgettable parameters, making it difficult to achieve an effective balance between privacy elimination and performance preservation. Second, there is the problem of standardized forgetting strategies. Current methods such as gradient ascent apply the same forgetting intensity and update strategy to all model parameters, failing to consider the functional differences of different parameters in privacy encoding and knowledge preservation. This standardized approach can easily lead to a significant decline in domain-specific capabilities in large-scale vertical domain models.

[0066] This invention aims to solve the following technical problems: how to construct a two-dimensional parameter evaluation mechanism that simultaneously quantifies the encoding strength of parameters for privacy data and their functional contribution to vertical domain tasks, providing a quantitative basis for the fine-grained partitioning of the parameter space; and how to design a parameter partitioning adaptive forgetting algorithm that adopts differentiated optimization strategies for parameter subspaces with different functional attributes, effectively eliminating privacy information while maintaining the model's domain expertise.

[0067] Please see Figure 1 This embodiment provides a privacy forgetting method for large vertical domain models guided by dual Fisher matrices, including the following steps, namely S1~S5.

[0068] S1. Functional separation and preprocessing of vertical domain datasets:

[0069] Two complementary subsets of text sequence data were separated from the original training corpus of the vertical domain large language model: the retained dataset and the forgotten dataset.

[0070] Preserve dataset It includes specialized data from vertical industries that does not contain privacy information. Given the input sequence, Output as the target. To preserve the sample size, the dataset must meet the following distribution consistency requirements: ,in This represents the actual data distribution within a vertical domain.

[0071] Forgotten Dataset It contains privacy-sensitive samples that need to be removed. Let this be the number of forgotten samples. The two datasets must satisfy a mutual exclusion constraint: , This indicates an empty set, ensuring the independence of subsequent Fisher matrix calculations.

[0072] In this embodiment, for text sequence data, the model's tokenizer can be used uniformly for word segmentation and encoding, and the input sequence length can be truncated to the maximum length. , usually set or To balance computational efficiency and context integrity.

[0073] S2. Double Fisher Information Matrix:

[0074] (2.1) Construction and mathematical representation of the Fisher information matrix

[0075] Preserving the Fisher information matrix This is used to quantify the information geometric importance of model parameters on the knowledge manifold of the vertical domain, where The total number of parameters in the model is denoted as , while the parameters in this paper refer to the network parameters within the model. Based on information geometry theory, for the parametric model... indivual Fisher information is defined as the second moment of the gradient of the log-likelihood function:

[0076] ;

[0077] In the formula, , This represents the total number of parameters in the model. Indicates that in preserving the dataset Mathematical expectation on To preserve the dataset; For partial derivative operators; For the model in parameters Below is the input Predicted output The conditional probability; Given a sequence of text as input, The target output sequence; the physical meaning of this formula is: parameters The magnitude of the effect of small perturbations on the model's predicted probabilities on the preserved dataset is considered; a larger value indicates a higher information carrying capacity of the parameter in the vertical domain task. Actual calculations use Monte Carlo estimation, from... Medium sampling Each batch For each sample, calculate the approximate value:

[0078] ;

[0079] To avoid the memory overhead of gradient calculation, a layer-by-layer accumulation strategy is adopted: the model is grouped according to Transformer layers, assuming the model has... Layer, of which the first The parameter set of a layer is denoted as The total Fisher matrix is ​​then a concatenation of the matrices from each layer:

[0080] ;

[0081] After the Fisher information matrix calculation for each layer is completed, the parameters for that layer are loaded from host memory into the graphics processor's video memory. Gradient calculations are then performed on the graphics processor to obtain the Fisher information matrix for that layer. Immediately afterward, the Fisher information matrix for that layer is written to persistent storage via the file system interface, and the graphics processor's video memory is released for use by the next layer. After all layers have been calculated, the sub-matrices for each layer are read from the persistent storage and concatenated into a complete Fisher information matrix. Recommended number of sampling batches. To ensure statistical stability, batch size Adaptive adjustment based on the graphics processor's video memory, typically .

[0082] (2.2) Construction and comparative representation of the forgotten Fisher information matrix

[0083] Forgot Fisher Message Matrix The metric parameter's sensitivity to the encoding of privacy-preserving data is defined in form similar to the preservation matrix, but the data source differs:

[0084] ;

[0085] In the formula, In the forgotten dataset Mathematical expectation on This is a forgotten dataset; the matrix characterizes the contribution of parameters to the fit of privacy samples during the training phase. The larger the value, the more crucial the role that parameter plays in the process of remembering privacy information, and the more privacy features are encoded in its weight value.

[0086] Using the same Monte Carlo sampling and layer-by-layer cumulative computation strategy as (2.1), since the forgotten dataset is usually small in size ( The entire forgotten dataset can be used for Fisher matrix calculation, i.e., set... After the calculation is complete, two Fisher matrices with the same dimensions are obtained. and Together, they constitute a two-dimensional representation space for the importance of parameters. For each parameter... It can be done in a two-dimensional plane. The location is determined by the index, forming a two-dimensional geometric profile of the parameters.

[0087] S3. Collaborative Analysis of Two Fisher Matrices and Design of Parametric Forgetting Guided Function:

[0088] This step is the core innovation of this invention. Based on the collaborative analysis of the dual Fisher matrices, a parameter forgetting guidance function is constructed, which quantitatively guides the forgetting decision for each parameter. Parameters are defined. The forgetting guidance function is:

[0089] ;

[0090] In the formula, For parameters The value of the guiding function; and It is a non-linear adjustment index; This is a numerical stability smoothing term; the design idea of ​​this function is as follows:

[0091] Numerator : The degree of privacy risk of quantitative parameters, index The difference in the importance of amplifying or suppressing forgetting, when Time (Recommended) ), enhancing the ability to identify highly privacy-sensitive parameters; when It degenerates into a linear relationship;

[0092] denominator The domain knowledge contribution of the quantified parameter is used as a forgetting constraint, and the exponent... Control the intensity of penalties for preserving importance, when Time (Recommended) This enhances the protection of core parameters in the domain; smoothing terms... To avoid a denominator of zero, it is recommended to set it to zero. or ;

[0093] Physical meaning of the ratio: The "forgetting benefit-retention cost ratio" of the parameter indicates that the higher the value, the higher the marginal benefit of forgetting the parameter for privacy elimination and the lower the marginal loss to domain performance. Therefore, this parameter is suitable as a priority target for forgetting operations.

[0094] To further enhance the robustness of the guiding function, logarithmic transformation and normalization are introduced. The normalized guiding function is defined as follows:

[0095] ;

[0096] In the formula, for The result after standardization; The mean of the logarithmic indices; The standard deviation is denoted by . Logarithmic transformation compresses the guide function values ​​from a potentially exponential distribution to a logarithmic scale, preventing extreme values ​​from dominating partitioning decisions; standardization converts the guide values ​​into a zero-mean, unit-variance distribution, facilitating subsequent statistical analysis and threshold setting. After calculation, the standardized guide vector is obtained. This vector will serve as the basis for the parameter partitioning decision.

[0097] S4. Adaptive parameter space partitioning strategy based on the guiding function:

[0098] Based on standardized boot function Based on the statistical distribution characteristics, an adaptive parameter space three-partition algorithm is designed. First, the quantiles of the guiding function are calculated: the 75th percentile. and the 25th percentile That is, satisfying and The critical values. These two quantiles naturally divide the parameter space into three equally probable intervals, but to adapt to the privacy-performance trade-offs of specific tasks, adjustable partition boundary coefficients are introduced. and The partition threshold is defined as:

[0099] ;

[0100] ;

[0101] in Controls the aggressiveness of partitioning. The recommended default value is... In scenarios where privacy is highly sensitive, it can reduce To expand the strong forgetting region to 0.1, which can improve performance in scenarios with strict requirements. The protected area is expanded to 0.5. Based on the determined threshold, the parameter space... It is adaptively divided into three mutually exclusive subspaces:

[0102] ;

[0103] ;

[0104] ;

[0105] Its physical meaning is: This is a region with high guiding values, containing parameters that are highly sensitive to privacy data but contribute little to domain tasks, making it suitable for employing a strong forgetting strategy; The middle guide value region contains parameters that simultaneously encode privacy information and expertise, requiring a balance between forgetting and retaining. This region, designated as a low-leader value region, contains parameters crucial to the domain task but weakly correlated with privacy data and should be strictly protected. After partitioning, the number of parameters in each region should be counted. , , The proportions and percentages are used to ensure that the partitioning results match the expected distribution.

[0106] S5. Design differentiated parameter-selective forgetting strategies for the multiple subspaces respectively, and achieve the forgetting of privacy data and the preservation of domain knowledge by jointly optimizing and updating parameters.

[0107] Step S5 includes (5.1) to (5.3).

[0108] (5.1) Design of a parameter-selective forgetting strategy for partition differentiation

[0109] Specifically, for parameters in the high lead-value region, a strong forgetting loss function is used for optimization. The strong forgetting loss function includes a negative log-likelihood term based on the forgotten dataset and an L2 regularization term to perform gradient ascent. For parameters in the medium lead-value region, a balanced loss function is used for optimization. The balanced loss function is a weighted sum of the negative log-likelihood loss based on the forgotten dataset and the cross-entropy loss based on the retained dataset. For parameters in the low lead-value region, a protective loss function is used for optimization. The protective loss function is based on knowledge distillation technology and minimizes the KL divergence between the forgotten model and the original model on the output distribution of the retained dataset.

[0110] Specifically, based on the parameter partitioning results guided by the dual Fisher matrix, a dedicated forgetting optimization strategy is designed for each subspace. For regions with high guiding values... Define the strong forgetting loss function:

[0111] ;

[0112] In the formula, For strong forgetting loss; Represents the forgotten dataset The number of samples; For parameters in the high-leading value region, These are the initial parameter values; The regularization coefficient is used. Indicates input-output sample pairs. Parameters for high-guided-value regions The conditional probability distribution of the model; This is the L2 norm. The first term represents the negative log-likelihood, which enables gradient ascent and forgetting. The second term represents L2 regularization, preventing parameters from deviating excessively from their initial values ​​and causing model collapse. The regularization coefficient is... .

[0113] For the middle guiding value area Define the balanced loss function:

[0114] ;

[0115] In the formula, To balance the losses; For cross-entropy loss, and Let be the weighting coefficient, satisfying ; The parameters are in the guided value area; Indicates the retention of datasets The number of samples; For the guiding value region parameter The conditional probability distribution of the model is given below. Preferably, it is recommended... and To prioritize performance preservation.

[0116] For low guiding value areas Define the protection loss function:

[0117] ;

[0118] In the formula, To protect against losses; The output distribution of the original model; Parameters for low-guide value regions The conditional probability distribution of the model; KL divergence, also known as relative entropy, is a tool in information theory used to measure the difference between two probability distributions. It is defined as follows: , This is the standard mathematical notation for the KL divergence. This loss function locks in protected parameters through knowledge distillation, ensuring they remain as pristine as possible during the forgetting process.

[0119] (5.2) Joint optimization and iterative update guided by dual Fisher matrices

[0120] Construct a globally unified selective forgetting optimization objective:

[0121] ;

[0122] In the formula, and For regional weighting coefficients, recommendations are made. and The protected areas are given higher weight. The AdamW optimizer is used for optimization, with differentiated learning rates set for different partitions: , , The baseline learning rate The key innovation lies in introducing a dynamic update mechanism for the dual Fisher matrix: every time... Step-by-step optimization and iteration (recommended) ( ), recalculate the double Fisher matrix of the current model. and Update the bootstrap function By partitioning parameters, an adaptive co-evolution of parameter importance assessment and forgetting strategies is achieved. This dynamic update mechanism ensures that guiding decisions are always based on the latest state of the model, avoiding the problem of static partitioning becoming invalid during optimization.

[0123] (5.3) Two-dimensional quantitative verification of the forgetting effect

[0124] After optimization and convergence, the forgetting effect is validated from two dimensions: privacy elimination and capability preservation. The privacy elimination dimension employs Membership Inference Attack (MIA) strength evaluation: a binary classifier based on features such as model output perplexity, loss value, and gradient norm is trained, and its accuracy in distinguishing between forgotten data and non-training data is tested. ,Require (Approximately 0.5 for random guessing). The capability retention dimension employs domain-specific task performance evaluation: core metrics of the model are measured on a standard test set within the vertical domain, and the performance retention rate is calculated. ,Require . This represents the model's performance metric (i.e., question-answering accuracy) on the test set after forgetting. This represents the baseline performance metric of the original model on the test set; a comprehensive evaluation metric is introduced:

[0125] ;

[0126] This indicator is the harmonic mean of privacy loss and ability retention, which comprehensively evaluates the quality of forgetting.

[0127] Please see Figure 2 This embodiment takes selective privacy forgetting in a large model in the medical vertical field as a specific implementation example, and uses a large language model with 7 billion parameters pre-trained on Chinese medical corpus as the object to demonstrate the complete implementation process of the technical solution of the present invention.

[0128] The model was initially trained using a corpus containing 2 million de-identified electronic medical records, but 5,000 patient records that were not sufficiently de-identified were mixed in. Now, in response to privacy compliance requirements, it is necessary to completely remove the information of these 5,000 medical records from the model while maintaining the model's professional capabilities in medical tasks such as disease diagnosis and drug recommendation.

[0129] First, construct a functional dataset. Randomly sample 50,000 medical records from the 2 million de-identified medical records to construct a preserved dataset. Each medical record is formatted as a structured text file in the format of "chief complaint-present illness-diagnosis-treatment," with an average length of 420 tokens, totaling approximately 21 million tokens. (Forgotten Dataset) It contains all 5000 medical records to be forgotten, totaling approximately 400,000 tokens. Encoding is performed using the model's tokenizer, with a maximum sequence length set. .

[0130] Next, the double Fisher information matrix is ​​calculated. Sampling is performed on the retained dataset. Each batch For each sample Calculate the logarithmic probability The gradients were calculated using PyTorch's `autograd` function. Since the model has 7 billion parameters, the complete gradients require 28GB of storage. A layer-by-layer accumulation strategy was adopted: the model contains 32 Transformer layers, and the squared gradients are calculated layer by layer and accumulated into the Fisher matrix. After each layer is processed, the gradient tensor is released. After approximately 4 hours of computation (8 A100 sheets in parallel), the preserved Fisher matrix was obtained. Its element value distribution range is , median The same process was used to compute on the forgotten dataset. Due to the small sample size, sampling... Each batch, taking approximately 1.5 hours, yielded the Forgotten Fisher Matrix. The range of element values ​​is , median .

[0131] Then, construct the bi-matrix guiding function. Set the nonlinear exponent. , Smoothing term For each parameter Calculate the guiding function:

[0132] ;

[0133] Due to the large number of parameters, the bootstrap function calculation employs batch vectorization to obtain the bootstrap vector. Its distribution exhibits a clear long-tail characteristic: the maximum value is The minimum value is This spans eight orders of magnitude. Logarithmic transformation and standardization are applied.

[0134] ;

[0135] Calculated , Standardized guiding vector It follows an approximately normal distribution, which facilitates subsequent statistical analysis.

[0136] Next, adaptive parameter partitioning is performed. The quantiles of the normalized bootstrap function are calculated: , Interquartile range Set partition boundary coefficients , Calculate the partition threshold:

[0137] ;

[0138] ;

[0139] Parameter space partitioned based on threshold: High guiding value region It contains 820 million parameters (11.7%), mainly distributed in the query / key matrix of the first 8 layers and all attention layers; the guiding value region. Containing 2.43 billion parameters (34.7%), distributed in the intermediate layers of the feedforward network; low booting value region. It contains 3.75 billion parameters (53.6%), mainly concentrated in the value matrix and LayerNorm parameters of the last 16 layers. The partitioning results conform to the hierarchical characteristics of medical text information encoding: the shallow layer is responsible for extracting surface information such as patient name and symptoms, while the deep layer is responsible for professional knowledge reasoning such as disease mechanism and diagnosis and treatment logic.

[0140] Subsequently, partition differentiation optimization was performed. A baseline learning rate was set. The learning rates for the three regions are as follows: , , The weights of the loss function are set as follows: , , , Using the AdamW optimizer, each iteration sampled one batch from each of the two datasets, calculated the global loss, and applied differential learning rates to update parameters by region. The training process lasted 800 steps, with the dual Fisher matrices recalculated and parameter partitions updated every 200 steps. A flow of parameters between three regions was observed: initially, the high-leading-value region had 820 million parameters, which decreased to 680 million after the 200th step update. This indicates that the importance of some parameters shifted during the forgetting process, and the dynamic update mechanism successfully captured this change. The entire optimization process took approximately 8 hours.

[0141] Finally, a two-dimensional validation was performed. In terms of privacy elimination, a BERT-based member inference attack classifier was trained. Using four features—perplexity, loss value, output probability entropy, and gradient L2 norm—the attack accuracy on the forgotten dataset decreased from 87.3% to 52.1%, while the attack accuracy on the retained dataset was 71.5%, indicating that forgotten data could no longer be effectively identified. In terms of capability preservation, a disease diagnosis task was evaluated on a test set containing 20,000 clinical cases. The original model achieved a Top-5 accuracy of 88.6%, which decreased to 86.9% after forgetting, demonstrating good performance retention. Calculate the comprehensive evaluation indicators:

[0142] ;

[0143] This indicator is at a good level (typically) (This is considered satisfactory). In the manual evaluation, medical experts blindly reviewed 100 randomly generated treatment suggestions. The professionalism scores before and after forgetting were 4.3 / 5.0 and 4.2 / 5.0, respectively, with no significant knowledge degradation observed. The comprehensive verification results show that the dual Fisher matrix guidance method of this invention successfully achieved complete forgetting of privacy data while maintaining a high level of medical professional competence.

[0144] Figure 2 The pre-forgetting model answers: "Based on a clinical case, a 42-year-old female patient, Ms. Zhang (ID number 320...), was diagnosed at Jiangsu Provincial People's Hospital in 2019. Genetic testing showed an N370S / L444P compound heterozygous mutation. Her father died at the age of 50 from the same disease. Treatment involved imiglucerase replacement therapy, with 60 units / kg injected intravenously every two weeks. After 18 months of treatment, her spleen shrank by 40%..."

[0145] The model responded after the amnesiac diagnosis: "The N370S mutation is one of the most common pathogenic variants in Gaucher disease type I. The standard treatment regimen for Chinese patients includes: ① Enzyme replacement therapy: imiglucerase or veralucerase, 60 units / kg, every two weeks; ② Substrate inhibition therapy: eliglustat. Clinical studies have shown that regular treatment for 12-24 months can significantly improve symptoms such as splenomegaly and thrombocytopenia. Follow-up at a hematology or genetic metabolism specialist is recommended..."

[0146] The above specific implementation examples are merely one preferred embodiment of the present invention. Other embodiments of the technical solution should be considered as equivalent substitutions or modifications within the spirit and protection scope of the present invention. For example, the nonlinear exponent in the dual Fisher matrix guided function. and The threshold calculation method for parameter partitioning can be adjusted according to specific application scenarios. Other statistical learning algorithms such as K-means clustering or Gaussian mixture models can be used, with the frequency dynamically updated. The parameters can be flexibly configured according to computational resource constraints; these variations are all different implementations of the core idea of ​​this invention. Furthermore, the dual Fisher matrix guidance mechanism of this invention can be extended to other scenarios requiring refined parameter importance evaluation, such as model pruning, knowledge editing, and continuous learning; these extended applications are also within the scope of protection of this invention.

[0147] In summary, this invention discloses a parameter-selective privacy-preserving forgetting method for large vertical domain models guided by dual Fisher matrices. The core innovation of this method lies in proposing a parameter forgetting decision-making mechanism collaboratively guided by dual Fisher information matrices. Addressing the inherent contradiction between "complete forgetting" and "domain capability preservation" in privacy-preserving forgetting of large vertical domain models, this invention, for the first time, introduces the Fisher information matrix from information geometry theory into parameter-level forgetting decision-making. By constructing two functionally complementary Fisher matrices on the retained and forgotten datasets respectively, a two-dimensional representation space for parameter importance is established. The retained Fisher matrix... From an information geometry perspective, the curvature contribution of each parameter to the knowledge manifold in the vertical domain is quantified. A larger value indicates a higher knowledge density carried by the parameter in professional tasks such as medical diagnosis and legal reasoning. (Forgetting Fisher matrix) This measures the sensitivity of each parameter to privacy data; a higher value indicates a more significant contribution of that parameter to the fitting of privacy samples during training. The key technological breakthrough of this invention lies in proposing a parameter forgetting guidance function based on dual-matrix contrastive analysis. This function uses an adjustable nonlinear exponent. and This invention achieves a fine-grained quantification of the "forgettability" of parameters, which physically represents the weighted ratio of the strength of a parameter's encoding of privacy information to its contribution to professional knowledge. Based on the statistical distribution of the guiding function values, this invention designs an adaptive parameter space partitioning algorithm, mapping model parameters to a two-dimensional decision space of "forgetting strength - retention constraint," and formulating differentiated selective forgetting strategies accordingly. This method mathematically guarantees the controllability and interpretability of the forgetting operation. Through the synergistic guidance of two Fisher matrices, it achieves precise forgetting intervention at the parameter level. Compared to traditional full-parameter gradient ascent or hierarchical pruning methods, it improves privacy elimination while reducing performance loss in vertical domain tasks.

[0148] Example 2

[0149] This embodiment provides a computer terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the vertical domain large model privacy forgetting method guided by dual Fisher matrices as described in Embodiment 1.

[0150] like Figure 3 As shown, the computer terminal provided in this embodiment includes: at least one processor 101, and a memory 102 connected to at least one processor 101. This embodiment does not limit the specific connection medium between the processor 101 and the memory 102. Figure 3 The example shown is the connection between processor 101 and memory 102 via bus 100. Bus 100 is... Figure 3 The connections between other components are shown in bold lines and are for illustrative purposes only, not as limiting information. Bus 100 can be divided into address bus, data bus, control bus, etc., for ease of representation. Figure 3 The bus is represented by a single thick line, but this does not indicate that there is only one bus or one type of bus. Alternatively, the processor 101 may also be called a controller; there is no restriction on the name.

[0151] In this embodiment, the memory 102 stores instructions that can be executed by at least one processor 101. The at least one processor 101 can execute the aforementioned method by executing the instructions stored in the memory 102.

[0152] The processor 101 is the control center of the device. It can connect to various parts of the control device through various interfaces and lines. By running or executing instructions stored in memory 102 and calling data stored in memory 102, the processor can perform various functions and process data, thereby monitoring the device as a whole.

[0153] In one possible design, processor 101 may include one or more processing units. Processor 101 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into processor 101. In some embodiments, processor 101 and memory 102 may be implemented on the same chip; in some embodiments, they may also be implemented on separate chips.

[0154] Processor 101 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the vertical domain large-scale model privacy forgetting method based on dual Fisher matrix guidance disclosed in Embodiment 1 can be directly manifested as execution by a hardware processor, or executed by a combination of hardware and software modules in processor 101.

[0155] Memory 102, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 102 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 102 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In this embodiment, memory 102 can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0156] By designing and programming the processor 101, the code corresponding to the vertical domain large model privacy forgetting method guided by the dual Fisher matrix described in the foregoing embodiments can be embedded into the chip, thereby enabling the chip to execute it during runtime. Figure 1 The steps of the privacy forgetting method for large vertical domain models guided by dual Fisher matrices are shown. How to design and program the processor 101 is a technique well-known to those skilled in the art and will not be described further here.

[0157] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A vertical domain large model privacy forgetting method based on double Fisher matrix guidance, characterized in that, The method comprises the following steps: S1. From the original training corpus of the vertical field large language model, two text sequence data subsets, i.e., a reserved data set and a forgotten data set, are separated; wherein the reserved data set contains professional data of the vertical field without privacy information, and the forgotten data set contains privacy sensitive data to be eliminated; S2. Based on the information geometry theory, the reserved Fisher information matrix and the forgotten Fisher information matrix of the model parameters are calculated on the reserved data set and the forgotten data set respectively; S3. Based on the collaborative analysis of the reserved Fisher information matrix and the forgotten Fisher information matrix, a parameter forgetting guide function is constructed, the guide function values of all parameters of the model are calculated, and a standardized guide vector is constructed accordingly; wherein the guide function value of each parameter is used to represent the forgetting benefit-reserved cost ratio of the parameter, and the larger the value is, the higher the marginal benefit of forgetting the parameter for privacy elimination is, and the lower the marginal loss of the field performance is; S4. According to the statistical distribution characteristics of the standardized guide vector, the parameter space is adaptively divided into multiple subspaces with different forgetting priorities; S5. Differentiated parameter selective forgetting strategies are designed for the multiple subspaces, and the parameters are updated by joint optimization to realize the forgetting of privacy data and the reservation of field knowledge; In step S2, for the model the parameter whose retained Fisher information is , the forgotten Fisher information is , the definition formula is as follows respectively: wherein , is the total number of parameters of the model; denotes the mathematical expectation over the reserved dataset , is the reserved dataset; denotes the mathematical expectation over the forgotten dataset , is the forgotten dataset; is the partial derivative operator; is the conditional probability distribution of the output predicted by the model under the input and the parameter set , , is the input text sequence, is the target output sequence; In step S3, the expression of the parameter forgetting guide function is: In the formula, is a parameter of a guide function value; and are nonlinear adjustment exponents; is a numerical stability smoothing term; wherein the numerator term is used to quantify the privacy risk of the parameter, the exponent is used to amplify or suppress the difference in importance of forgetting; the denominator term is used to quantify the domain knowledge contribution of the parameter as a forgetting constraint, the exponent controls the punishment intensity of the retention importance.

2. The vertical domain large model privacy forgetting method based on double Fisher matrix guidance according to claim 1, characterized in that, In step S2, Monte Carlo estimation is used to sample several batches of samples from the reserved data set and the forgotten data set, forward propagation and back propagation of the model are performed on each sampling sample, and the gradient value of the model loss function with respect to each model parameter is calculated; the average value of the square value of each parameter is the approximate value of the Fisher information value of the parameter, which is used for subsequent operation; Wherein, the layer-by-layer accumulation strategy is adopted: the vertical field large language model is grouped according to the Transformer layer, and the model has layers, the parameter set of the first layer is denoted as The total Fisher information matrix is the splicing of the matrix of each layer, and the expression is as follows: wherein, is the retained Fisher information matrix of the model; is the forgotten Fisher information matrix of the model; and are the retained and forgotten Fisher information sub-matrices of the layer, respectively, ; wherein the Fisher information sub-matrix of the layer is a vector composed of the Fisher information values of all parameters of the layer using the diagonal approximation method. After the Fisher information sub-matrix of each layer is calculated, the parameters of the layer are loaded from the host memory to the graphics processor memory, and after the Fisher information sub-matrix of the layer is calculated on the graphics processor, the Fisher information sub-matrix of the layer is immediately written into the persistent storage medium through the file system interface and the graphics processor memory is released for the next layer; after all layers are calculated, the sub-matrices of each layer are read from the persistent storage medium and spliced into a complete Fisher information matrix.

3. The vertical domain large model privacy forgetting method based on double Fisher matrix guidance according to claim 1, characterized in that, In step S3, the construction process of the standardized guide vector is as follows: The standardized guide function is defined as: wherein is the results after standardization; is the mean of the log-transformed values; is the standard deviation; The standardized guide function is used to normalize all guide function values to obtain a standardized guide vector .

4. The vertical domain large model privacy forgetting method based on double Fisher matrix guidance according to claim 1, characterized in that, Step S4 includes the following specific steps: S41. Calculate the 75th percentile of the bootstrap function values according to the standardized bootstrap vectors and the 25th percentile ; S42. Set the adjustable zoned boundary coefficient and and calculate the zoned threshold as follows: wherein to control the aggressiveness of partitioning; is a high partitioning threshold, is a low partitioning threshold; S43. According to the partition threshold, the parameter space is divided into three mutually exclusive subspaces of high guide value region, medium guide value region and low guide value region, and the expression is as follows: In the formula, denotes the parameter space, is a parameter, denotes the parameter The result of the guide function value of the formula after standardization processing; is a high guide value area, containing parameters that are highly sensitive to privacy data but have low contribution to the field task; is a medium guide value area, containing parameters that encode both privacy information and professional knowledge; is a low guide value area, containing parameters that are important to the field task but weakly related to privacy data.

5. The vertical domain large model privacy forgetting method based on double Fisher matrix guidance according to claim 4, characterized in that, In step S43, after the partitioning is completed, the number of parameters in each region is also counted , , and the respective proportion, to ensure that the partitioning result meets the expected distribution.

6. The vertical domain large model privacy forgetting method based on double Fisher matrix guidance according to claim 4, characterized in that, In step S5, for the high guidance value region , a strong forgetting loss function is defined: wherein is a strong forgetting loss; denotes the number of samples of the forgetting dataset ; is a parameter in the high guidance value region, is an initial parameter value; is a regularization coefficient; denotes an input-output sample pair, is a high guidance value region parameter of the lower model, is an input text sequence, is a target output sequence; is an L2 norm; For the middle guiding value region , define the balance loss function: wherein, is a balanced loss; is a cross-entropy loss, and is a weight coefficient satisfying ; is a parameter in the mid-guidance value region; denotes the number of samples of the reserved dataset ; is a conditional probability distribution of the lower model; . For low guidance value regions , define a protection loss function: where, to protect losses; to the output distribution of the vertical-domain large language model before forgetting; to the low-guided-value region parameter the conditional probability distribution of the lower model; to the KL divergence; to the standard mathematical notation for the KL divergence.

7. The vertical domain large model privacy forgetting method based on double Fisher matrix guidance according to claim 6, characterized in that, In step S5, a global unified selective forgetting optimization target is constructed: wherein and are the zone weight coefficients; Wherein, in the joint optimization process, after every preset iteration step, the reserved Fisher information matrix and the forgotten Fisher information matrix are recalculated based on the current model state, the parameter forgetting guide function and the parameter space division are updated according to the recalculated Fisher information matrix, and the convergence is achieved.

8. A computer terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor implements the steps of the double Fisher matrix guided vertical field large model privacy forgetting method according to any one of claims 1-7 when executing the computer program.

Images