Vertical field large model privacy forgetting method and terminal based on double Fisher matrix guidance

Abstract

The invention relates to the technical field of artificial intelligence and privacy protection, and discloses a vertical field large model privacy forgetting method and terminal based on double Fisher matrix guidance, and the method comprises the steps: separating two text sequence data subsets from an original training corpus of a vertical field large language model, namely a reserved data set and a forgetting data set; the reserved data set contains vertical field professional data which does not contain privacy information, and the forgotten data set contains privacy sensitive data to be eliminated. Based on an information geometry theory, calculating a reserved Fisher information matrix and a forgotten Fisher information matrix of model parameters on the reserved data set and the forgotten data set respectively; based on collaborative analysis of a reserved Fisher information matrix and a forgotten Fisher information matrix, a parameter forgotten guide function is constructed, guide function values of all parameters of a model are calculated, and a standardized guide vector is constructed according to the guide function values. According to the method, refined balance of privacy forgetting of the large language model in two levels of privacy elimination and capability keeping is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and privacy protection technology, in particular to a vertical field large model privacy forgetting method and terminal based on double Fisher matrix guidance. BACKGROUND

[0002] With the deep penetration of artificial intelligence technology, large-scale language models have become the core technology base of key vertical fields such as medical intelligent diagnosis, legal intelligent consultation, financial risk prediction, and public opinion analysis. These field-specific large language models, through continuous pre-training and supervised fine-tuning on industry corpus of tens of billions to hundreds of billions of tokens, have obtained deep understanding of professional knowledge and complex reasoning ability, and can provide auxiliary diagnosis suggestions for clinicians, generate legal opinions for lawyers, and predict market trends for financial analysts. However, in order to achieve the performance level of practicality, these vertical field large models must ingest massive real data including medical electronic medical record system data, court judgment documents, social media user post records, and enterprise internal operation documents during the training phase. These training corpus inevitably contain a large amount of personal privacy information, such as patient genetic test reports and medication records, litigant identity information and family relationships, social media user geolocation trajectory and work unit, and other sensitive data. Due to the inherent memory effect of deep neural networks, these privacy information will be encoded in the model's billion-level parameter tensor in a distributed manner, forming a deep-level privacy leakage risk. During the model inference phase, when users conduct exploratory queries through carefully designed prompt words, the model may fragment and associate the privacy information remembered during the training process across texts and semantic splicing, outputting highly sensitive privacy data combinations containing real names, ID numbers, medical records, social relationship networks, etc., violating relevant laws and regulations on data minimization and "right to be forgotten", and posing potential risks to personal privacy rights and data security.

[0003] To address the privacy data removal problem in large language models, the academic and industrial communities have proposed the Machine Unlearning technology paradigm, which aims to enable a trained model to selectively "forget" the influence of specific training samples, as if these data had never been involved in training. Current mainstream model forgetting methods mainly include complete retraining, gradient ascent forgetting, and influence function approximation. The complete retraining method physically deletes privacy data from the training corpus and then trains the model from scratch. Although this method can completely eliminate privacy influence, it requires thousands of GPU cards to perform a single retraining, which consumes thousands of GPU card days of computing resources, takes weeks of time, and costs hundreds of thousands of yuan. Moreover, each new privacy deletion request requires repeating the process, limiting its practical application in engineering practice. The gradient ascent forgetting method actively increases the model's prediction loss by performing gradient ascent optimization on the data to be forgotten, but this method lacks a fine-grained distinction of parameter importance, causing indiscriminate disturbance to the model parameter space and leading to severe degradation of performance in core vertical tasks such as medical diagnosis and legal text retrieval, with accuracy reduction of up to 20-40%. The influence function method attempts to locate the parameters that need to be corrected by calculating the influence of training samples on model parameters, but this method requires the calculation of the inverse of the Hessian matrix, which has a computational complexity of O(n³) for large models with billions of parameters, making it infeasible in practical applications. More critically, existing methods lack a verifiable guarantee mechanism for forgetting effects, making it impossible to prove that privacy data has been completely forgotten rather than just superficially suppressed, and there is a security risk of still being able to extract privacy information through adversarial prompt attacks. SUMMARY

[0004] To address the technical problem of the difficulty of balancing privacy removal and capability preservation in existing large language model privacy forgetting methods, thereby limiting the application of models in some vertical fields, the present invention provides a vertical field large model privacy forgetting method based on double Fisher matrix guidance.

[0005] To achieve the above-mentioned purposes, the present invention provides the following technical solutions: The present invention discloses a vertical field large model privacy forgetting method based on double Fisher matrix guidance, comprising the following steps, i.e. S1-S5.

[0006] S1. From the original training corpus of the vertical domain large language model, two subsets of text sequence data are separated, namely the retained dataset and the forgotten dataset; wherein, the retained dataset contains vertical domain professional data without privacy information, and the forgotten dataset contains privacy-sensitive data to be eliminated.

[0007] S2. Based on information geometry theory, calculate the retained Fisher information matrix and the forgotten Fisher information matrix of the model parameters on the retained dataset and the forgotten dataset, respectively.

[0008] S3. Based on the synergistic analysis of the retained Fisher information matrix and the forgotten Fisher information matrix, construct a parameter forgetting guidance function, calculate the guidance function values ​​of all parameters of the model, and construct a standardized guidance vector accordingly; wherein, the guidance function value of each parameter is used to characterize the forgetting benefit-retention cost ratio of that parameter, the larger the value, the higher the marginal benefit of forgetting that parameter for privacy elimination, and the lower the marginal loss to domain performance.

[0009] S4. Based on the statistical distribution characteristics of the standardized guiding vector, the parameter space is adaptively divided into multiple subspaces with different forgetting priorities.

[0010] S5. Design differentiated parameter-selective forgetting strategies for the multiple subspaces respectively, and achieve the forgetting of privacy data and the preservation of domain knowledge by jointly optimizing and updating parameters.

[0011] As a further improvement to the above scheme, in step S2, for model number... Parameters It retains Fisher information as Forgot Fisher information The formulas are defined as follows: ; ; In the formula, , This represents the total number of parameters in the model. Indicates that in preserving the dataset Mathematical expectation on To preserve the dataset; In the forgotten dataset Mathematical expectation on Forgotten dataset; For partial derivative operators; For the model in the parameter set Based on the input Predicted output The conditional probability distribution, , input text sequence, target output sequence.

[0012] As a further improvement of the above scheme, in step S2, Monte Carlo estimation is used to sample a plurality of batches of samples from the retained dataset and the forgotten dataset respectively, forward propagation and back propagation of the model are performed on each sampled sample, and the gradient value of the model loss function with respect to each model parameter is calculated; the average value of the square value of the gradient of each parameter over all sampled samples is the approximate value of the Fisher information value of the parameter, which is used for subsequent operations.

[0013] Where, the layer-by-layer accumulation strategy is adopted: the vertical field large language model is grouped according to the Transformer layer, and the model has a total of L layers, the parameter set of the i-th layer is denoted as The total Fisher information matrix is the splicing of the matrices of each layer, and the expression is as follows: ; ; In the formula, is the retained Fisher information matrix; is the forgotten Fisher information matrix; and are the retained Fisher information sub-matrix and the forgotten Fisher information sub-matrix of the i-th layer, respectively, ; wherein the Fisher information sub-matrix of the i-th layer is a vector composed of the Fisher information values of all parameters of the layer by using the diagonal approximation method. After the Fisher information sub-matrix of each layer is calculated, the parameters of the layer are loaded from the host memory to the graphics processor memory, and after the Fisher information sub-matrix of the layer is calculated by performing gradient calculation on the graphics processor, the Fisher information sub-matrix of the layer is immediately written to the persistent storage medium through the file system interface and the graphics processor memory is released for the next layer; after all layers are calculated, the sub-matrices of each layer are read from the persistent storage medium and spliced into a complete Fisher information matrix.

[0014] As a further improvement of the above scheme, in step S3, the expression of the parameter forgetting guide function is:

[0015] ; ; In the formula, is the guide function value of the parameter ; and ​​​is a non-linear adjustment exponent; is a numerical stability smoothing term; wherein the numerator is the privacy risk degree of the quantization parameter, the exponent is the difference for amplifying or suppressing the importance of forgetting; the denominator is the domain knowledge contribution degree of the quantization parameter as a forgetting constraint, the exponent is the penalty strength for controlling the retention importance.

[0016] As a further improvement of the above scheme, in step S3, the construction process of the standardized guide vector is as follows: Define the standardized guide function: ; wherein, is the mean value of the logarithmic guide value; is the result after standardization processing; is the mean value of the logarithmic guide value; is the standard deviation.

[0017] The standardized guide function is used to standardize all guide function values to obtain the standardized guide vector .

[0018] As a further improvement of the above scheme, step S4 includes the following specific steps, i.e., S41-S43.

[0019] S41. Calculate the 75th percentile and the 25th percentile of the guide function value according to the standardized guide vector.

[0020] S42. Set the adjustable partition boundary coefficient and , and calculate the partition threshold according to the following formula: ; ; wherein, is used to control the aggressiveness of the partition; is the high partition threshold, is the low partition threshold.

[0021] S43. According to the partition threshold, divide the parameter space into three mutually exclusive subspaces of high guide value region, medium guide value region and low guide value region, expressed as follows: ; ; ; wherein, denote the parameter space, for parameters, denote the parameters whose guide function values are processed by standardization; denote the high guide value region, containing parameters that are highly sensitive to privacy data but have low contribution to the field task; denote the medium guide value region, containing parameters that encode both privacy information and professional knowledge; denote the low guide value region, containing parameters that are important to the field task but weakly related to privacy data.

[0022] As a further improvement of the above scheme, in step S43, after partitioning is completed, the number of parameters in each region , , and the respective proportion are also counted to ensure that the partition result meets the expected distribution.

[0023] As a further improvement of the above scheme, in step S5, for the high guide value region , define a strong forgetting loss function: ; where, is the strong forgetting loss; denotes the number of samples in the forgetting dataset ; is the parameter in the high guide value region, is the initial parameter value; is the regularization coefficient; denotes an input-output sample pair, is the conditional probability distribution of the model under the high guide value region parameter ; is the input text sequence, is the target output sequence; is the L2 norm.

[0024] For the medium guide value region , define a balanced loss function: ; where, is the balanced loss; is the cross-entropy loss, and are weight coefficients satisfying ; is the parameter in the medium guide value region; denotes the number of samples in the reserved dataset ; is the conditional probability distribution of the model under the medium guide value region parameter .

[0025] For low guidance value area , define the protection loss function: ; In the formula, is the protection loss; is the output distribution of the vertical field large language model before forgetting; is the conditional probability distribution of the low guidance value area parameter under the model; is the KL divergence; is the standard mathematical symbol of KL divergence.

[0026] As a further improvement of the above scheme, in step S5, a globally unified selective forgetting optimization target is constructed: ; In the formula, and are area weight coefficients.

[0027] Wherein, in the joint optimization process, after every preset number of iterations, the reserved Fisher information matrix and the forgetting Fisher information matrix are recalculated based on the current model state, and the parameter forgetting guidance function and the parameter space division are updated according to the recalculated Fisher information matrix until convergence.

[0028] The application also discloses a computer terminal, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and characterized in that when the processor executes the computer program, the steps of the vertical field large model privacy forgetting method based on double Fisher matrix guidance are realized.

[0029] Compared with the prior art, the application has the following beneficial effects: 1. The double Fisher matrix guiding method of the present application realizes double-dimensional quantification analysis of the parameter space of a large language model. By calculating the Fisher information matrix on the reserved dataset and the forgotten dataset respectively, the present application can simultaneously depict the functional attributes of each parameter: the reserved Fisher information matrix reflects the contribution of the parameter to the vertical field task, and the forgotten Fisher information matrix reflects the encoding strength of the parameter to the private data. The parameter forgetting guiding function constructed based on the two can give each parameter a quantifiable forgetting priority, representing the forgetting benefit and retention cost ratio of the parameter, solving the problem of single parameter importance evaluation in the prior art. The embodiment verification shows that the method can accurately identify the parameter subspace which is highly sensitive to private data but has low contribution to the field task. The directed forgetting of these parameters makes it difficult for private data to be identified by member inference attacks, while maintaining the performance of the model in the vertical field task, which is better than the performance of the traditional gradient ascent method.

[0030] 2. The parameter three-partition adaptive forgetting algorithm designed by the present application effectively solves the contradiction between forgetting strength and performance preservation. According to the statistical distribution characteristics of the standardized guiding vector, the system automatically divides the parameter space into three subspaces of high guiding value region, medium guiding value region and low guiding value region, and configures a differentiated loss function for each partition: the high guiding value region adopts strong forgetting loss to realize gradient ascent forgetting, the medium guiding value region adopts balanced loss to combine forgetting and retention targets, and the low guiding value region adopts protection loss to lock the output distribution through KL divergence. This partition strategy enables the algorithm to accurately update the parameters related to private data while protecting the key parameters carrying domain knowledge. In addition, the dynamic updating mechanism introduced by the present application re-calculates the double Fisher matrix and updates the parameter partition every preset iteration step, ensuring that the forgetting strategy is always based on the latest state of the model, and the parameter partition ratio can be flexibly adjusted according to specific business requirements.

[0031] 3、The privacy forgetting method realized by the present application can solve the timeliness technical problem of vertical field large models in privacy regulation compliance. For privacy-sensitive vertical business scenarios such as medical diagnosis and legal consultation, a large language model may encode tens of thousands of patient medical records or thousands of confidential legal documents during training. When individual data subjects exercise the right to delete, the traditional complete retraining method requires retraining the entire model from zero, which takes a long time for a single retraining cycle, and each new deletion request requires repeating the process, resulting in a backlog of privacy deletion requests that cannot be processed in a timely manner in actual engineering. The parameterized forgetting method of the present application shortens the execution time of a single forgetting operation to the order of hours by precisely positioning and updating the model parameter subspace, and supports batch processing of multiple deletion requests, so that the rights of data subjects can be responded to in a timely manner. This technology breakthrough provides compliance assurance for deploying large language models in strictly regulated vertical industries such as healthcare, finance, and law, and solves the conflictive technical problem between privacy protection regulations and AI technology application, enabling the application of large models in high privacy-sensitive fields to change from compliance unfeasibility to technical feasibility. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 The flowchart of the privacy forgetting method of the vertical field large model based on the double Fisher matrix guide in the embodiment 1 of the present application.

[0033] Figure 2 The selective privacy forgetting process example diagram of the medical vertical field large model in the embodiment 1 of the present application.

[0034] Figure 3 The structural schematic diagram of the computer terminal in the embodiment 2 of the present application. DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0036] Embodiment 1 The existing privacy forgetting technology of large language models has the following limitations. First, the single-dimensional problem of parameter importance evaluation. Existing methods such as continuous learning technology based on elastic weight consolidation mainly focus on protecting parameters important to the original task, while the model forgetting scenario needs to address the dual constraints of identifying parameters that encode privacy information and identifying parameters that carry domain knowledge. Since some parameters may have both dual functional properties, single-dimensional importance evaluation cannot accurately define the boundaries of forgettable parameters, making it difficult to balance privacy elimination and performance preservation in forgetting operations. Second, the problem of uniform processing of forgetting strategies. Current methods such as gradient ascent use the same forgetting strength and update strategy for all model parameters, without considering the functional differences of different parameters in privacy encoding and knowledge retention. This uniform processing approach can significantly reduce domain expertise in vertical domain large models.

[0037] The present application aims to solve the following technical problems: how to construct a two-dimensional parameter evaluation mechanism to quantify the encoding strength of parameters on privacy data and the functional contribution of parameters to vertical domain tasks, providing a quantitative basis for fine-grained partitioning of parameter space; how to design a parameter partition adaptive forgetting algorithm that uses differentiated optimization strategies for parameter subspaces with different functional attributes, effectively eliminating privacy information while maintaining the domain expertise of the model.

[0038] Please refer to Figure 1 The present embodiment provides a vertical domain large model privacy forgetting method based on double Fisher matrix guidance, comprising the following steps S1-S5.

[0039] S1. Functional separation and preprocessing of vertical domain dataset: From the original training corpus of the vertical domain large language model, two functionally complementary text sequence data subsets, the retention dataset and the forgetting dataset, are separated.

[0040] Retention dataset Contains vertical domain professional data without privacy information. Among them is the input sequence, is the target output, is the number of retention samples. This dataset must meet the distribution consistency requirement: , is the true data distribution of the vertical domain.

[0041] Forgetting dataset Contains privacy-sensitive samples that need to be eliminated. is the number of forgetting samples. The two datasets must meet the mutual exclusivity constraint: , represents an empty set, ensuring the independence of subsequent Fisher matrix calculations.

[0042] In this embodiment, for text sequence data, tokenizer of the model is uniformly used for word segmentation coding, and input sequence length is truncated to the maximum length , usually set or to balance the calculation efficiency and context integrity.

[0043] S2. Double Fisher information matrix: (2.1) Construction and mathematical representation of Fisher information matrix Fisher information matrix is used to quantify the information geometric importance of model parameters on the vertical domain knowledge manifold, where is the total number of model parameters, and the parameters in this paper are network parameters in the model. Based on the information geometric theory, for the parameter model, the Fisher information of the th parameter is defined as the second moment of the gradient of the log-likelihood function: ; where , is the total number of model parameters; denotes the mathematical expectation on the reserved data set , is the reserved data set; is the partial derivative operator; is the conditional probability of the model predicting the output under the input under the parameter ; is the input text sequence, is the target output sequence; the physical meaning of this formula is that the influence of a small perturbation of the parameter on the prediction probability of the model on the reserved data set is greater, which indicates that the information carrying capacity of the parameter in the vertical domain task is higher. In actual calculation, Monte Carlo estimation is used, and samples batches, each batch has samples, and the approximate value is calculated as: ; In order to avoid the memory overhead of gradient calculation, the layer-by-layer accumulation strategy is adopted: the model is grouped according to the Transformer layer, and the model has layers, where the parameter set of the th layer is denoted as , and the total Fisher matrix is the splicing of the matrices of each layer: ; After the calculation of the Fisher information sub-matrix of each layer is completed, the parameters of the layer are loaded from the host memory to the graphics processor video memory, and after the gradient calculation is performed on the graphics processor to obtain the Fisher information sub-matrix of the layer, the Fisher information sub-matrix of the layer is immediately written into the persistent storage medium through the file system interface and the graphics processor video memory is released for the next layer; after the calculation of all layers is completed, the sub-matrices of each layer are read from the persistent storage medium and spliced into a complete Fisher information matrix. Recommended sampling batch number To ensure statistical stability, batch size According to the adaptive adjustment of the graphics processor video memory, usually .

[0044] (2.2) Construction and comparative representation of forgetting Fisher information matrix Forgetting Fisher information matrix Measures the encoding sensitivity of the parameter to the private data, which is defined and remains consistent in form with the retention matrix but has a different data source: ; In the formula, represents the mathematical expectation on the forgetting data set , is the forgetting data set; the matrix describes the fitting contribution of the parameter to the private sample during the training stage, The greater the value, the more critical the role of the parameter in remembering private information, and the more private features are encoded in its weight value.

[0045] The same Monte Carlo sampling and layer-by-layer accumulation calculation strategy as (2.1) is adopted, and since the forgetting data set is usually small in size (N ), the entire forgetting data set can be used for Fisher matrix calculation, i.e. setting After calculation, two Fisher matrices with the same dimensions and are obtained, which together constitute a two-dimensional representation space of parameter importance. For each parameter , its position can be marked in the two-dimensional plane , forming a two-dimensional information geometric portrait of the parameter.

[0046] S3. Dual Fisher matrix collaborative analysis and parameter forgetting guide function design: This step is the core innovative link of the present application. Based on the collaborative analysis of the dual Fisher matrix, a parameter forgetting guide function is constructed, which quantitatively guides the forgetting decision of each parameter. The forgetting guide function of parameter is defined as: ; where, is the parameter of the guide function value; and are nonlinear adjustment exponents; is a numerical stability smoothing term; the design idea of this function is as follows: the numerator term : the privacy risk degree of the quantized parameter, the exponent is used to amplify or suppress the importance of forgetting, when (recommended ), the identification ability of high privacy-sensitive parameters is enhanced; when , it degenerates to a linear relationship; the denominator term : the domain knowledge contribution degree of the quantized parameter as a forgetting constraint, the exponent controls the punishment strength of retaining importance, when (recommended ), the protection strength of domain core parameters is enhanced; the smoothing term avoids the denominator being zero, and it is recommended to be set to or ; The physical meaning of the ratio is: It represents the "forgetting benefit-retaining cost ratio" of the parameter, the larger the value, the higher the marginal benefit of forgetting the parameter for privacy elimination, and the lower the marginal loss for domain performance, so the parameter is suitable as the priority target of the forgetting operation.

[0047] To further enhance the robustness of the guide function, logarithmic transformation and normalization processing are introduced. Define the standardized guide function: ; where, is the result after standardization; is the mean of the logarithmic guide value; is the standard deviation. The logarithmic transformation compresses the guide function value from the possible exponential distribution to the logarithmic scale, avoiding the dominance of extreme values in partition decision-making; the standardization processing converts the guide value into a distribution with zero mean and unit variance, facilitating subsequent statistical analysis and threshold setting. After calculation, the standardized guide vector is obtained, which will serve as the decision basis for parameter partitioning.

[0048] S4. Adaptive parameter space partitioning strategy based on guide function: Based on the statistical distribution characteristics of the standardized guide function , an adaptive parameter space tri-partition algorithm is designed. First, calculate the quantile of the guide function: the 75th percentile and the 25th percentile That is, satisfying and The critical values. These two quantiles naturally divide the parameter space into three equally probable intervals, but to adapt to the privacy-performance trade-offs of specific tasks, adjustable partition boundary coefficients are introduced. and The partition threshold is defined as: ; ; in Controls the aggressiveness of partitioning. The recommended default value is... In scenarios where privacy is highly sensitive, it can reduce To expand the strong forgetting region to 0.1, which can improve performance in scenarios with strict requirements. The protected area is expanded to 0.5. Based on the determined threshold, the parameter space... It is adaptively divided into three mutually exclusive subspaces: ; ; ; Its physical meaning is: This is a region with high guiding values, containing parameters that are highly sensitive to privacy data but contribute little to domain tasks, making it suitable for employing a strong forgetting strategy; The middle guide value region contains parameters that simultaneously encode privacy information and expertise, requiring a balance between forgetting and retaining. This region, designated as a low-leader value region, contains parameters crucial to the domain task but weakly correlated with privacy data and should be strictly protected. After partitioning, the number of parameters in each region should be counted. , , The proportions and percentages are used to ensure that the partitioning results match the expected distribution.

[0049] S5. Design differentiated parameter-selective forgetting strategies for the multiple subspaces respectively, and achieve the forgetting of privacy data and the preservation of domain knowledge by jointly optimizing and updating parameters.

[0050] Step S5 includes (5.1) to (5.3).

[0051] (5.1) Design of a parameter-selective forgetting strategy for partition differentiation Specifically, for parameters in the high lead-value region, a strong forgetting loss function is used for optimization. The strong forgetting loss function includes a negative log-likelihood term based on the forgotten dataset and an L2 regularization term to perform gradient ascent. For parameters in the medium lead-value region, a balanced loss function is used for optimization. The balanced loss function is a weighted sum of the negative log-likelihood loss based on the forgotten dataset and the cross-entropy loss based on the retained dataset. For parameters in the low lead-value region, a protective loss function is used for optimization. The protective loss function is based on knowledge distillation technology and minimizes the KL divergence between the forgotten model and the original model on the output distribution of the retained dataset.

[0052] Specifically, based on the parameter partitioning results guided by the dual Fisher matrix, a dedicated forgetting optimization strategy is designed for each subspace. For regions with high guiding values... Define the strong forgetting loss function: ; In the formula, For strong forgetting loss; Represents the forgotten dataset The number of samples; For parameters in the high-leading value region, These are the initial parameter values; The regularization coefficient is used. Indicates input-output sample pairs. Parameters for high-guided-value regions The conditional probability distribution of the model; This is the L2 norm. The first term represents the negative log-likelihood, which enables gradient ascent and forgetting. The second term represents L2 regularization, preventing parameters from deviating excessively from their initial values ​​and causing model collapse. The regularization coefficient is... .

[0053] For the middle guiding value area Define the balanced loss function: ; In the formula, To balance the losses; For cross-entropy loss, and For the weighting coefficients, satisfying ; The parameters are in the middle guide value area; Indicates the retention of datasets The number of samples; For the middle guiding value region parameter The conditional probability distribution of the model is given below. Preferably, it is recommended... and To prioritize performance preservation.

[0054] For low guiding value areas , define the protection loss function: ; where, is the protection loss; is the output distribution of the original model; is the low-guided value region parameter conditional probability distribution of the lower model; is the KL divergence, also known as relative entropy, which is a tool in information theory for measuring the difference between two probability distributions, defined as , is the standard mathematical notation for KL divergence. This loss function locks the protection zone parameters through knowledge distillation, so that they remain as original as possible during the forgetting process.

[0055] (5.2) Joint optimization and iterative update guided by double Fisher matrix Construct a global unified selective forgetting optimization objective: ; where, and are regional weight coefficients, recommended and , giving higher weight to the protection zone. The optimization uses the AdamW optimizer, with different learning rates for different partitions: , , , where the base learning rate . The key innovation is the introduction of a dynamic update mechanism for the double Fisher matrix: every optimization iterations (recommended ), the double Fisher matrix and of the current model are recalculated, and the guide function and parameter partition are updated, realizing the adaptive co-evolution of parameter importance evaluation and forgetting strategy. This dynamic update mechanism ensures that the guide decision is always based on the latest state of the model, avoiding the problem of static partition failure during optimization.

[0056] (5.3) Dual-dimensional quantitative verification of forgetting effect

[0057] After optimization convergence, the forgetting effect is verified from two dimensions: privacy elimination and ability preservation. The privacy elimination dimension uses the member inference attack (MIA) strength evaluation: a binary classifier based on model output perplexity, loss value, gradient norm, etc. is trained to test its accuracy in distinguishing forgotten data and non-training data , requiring (approximately random guess of 0.5). The capability preservation dimension employs domain task performance evaluation: measure the core metrics of the model on the standard test set of the vertical, compute the performance preservation rate , requires . to represent the performance indicator value of the model after forgetting on the test set (i.e., the accuracy of the question and answer), to represent the benchmark value of the performance indicator of the original model on the test set; introduce a comprehensive evaluation index: ; The index is the harmonic mean of privacy elimination and capability preservation, which comprehensively evaluates the forgetting quality.

[0058] For reference Figure 2 , the embodiment takes selective privacy forgetting of a medical vertical domain large model as a specific implementation example, takes a 7 billion parameter large language model pre-trained on Chinese medical corpus as an object, and demonstrates the complete implementation process of the technical solution of the application.

[0059] The model uses a corpus containing 2 million de-identified electronic medical records for initial training, but 5,000 patient medical records that are not fully desensitized are mixed in, and now needs to respond to privacy compliance requirements to completely eliminate the information of the 5,000 medical records from the model, while maintaining the professional capabilities of the model in medical tasks such as disease diagnosis and drug recommendation.

[0060] First, build a functional data set. Randomly sample 50,000 from the 2 million de-identified medical records to build a retention data set , each medical record is formatted as a structured text of "chief complaint-past medical history-diagnosis-treatment", with an average length of 420 tokens, totaling about 21 million tokens. The forgetting data set contains all 5,000 medical records to be forgotten, totaling about 400,000 tokens. Encode using the tokenizer of the model, set the maximum sequence length .

[0061] Second, calculate the double Fisher information matrix. On the retention data set, sample batches, each batch samples, calculate the log probability for each sample and calculate the gradient through PyTorch autograd. Since the model has 7 billion parameters, the complete gradient requires 28 GB of storage, and a layer-by-layer accumulation strategy is adopted: the model contains 32 layers of Transformers, and the gradient square is calculated and accumulated to the Fisher matrix layer by layer, and the gradient tensor is released after each layer is processed. After about 4 hours of calculation (8 A100 parallel), the retention Fisher matrix with element value ranging from to . The same procedure was applied on the forgetting dataset, with batches sampled due to the small sample size, taking about 1.5 hours, resulting in the forgetting Fisher matrix with element value ranging from to .

[0062] Then, the dual-matrix guiding function was constructed. The non-linear exponent , and smoothing term were set, and the guiding function was calculated for each parameter : ; Due to the large number of parameters, the guiding function was calculated using batch vectorization, resulting in the guiding vector , which showed a clear long-tail feature: the maximum value was , and the minimum value was , spanning 8 orders of magnitude. Log transformation and standardization were applied: ; resulting in , , and the standardized guiding vector conformed to an approximate normal distribution, facilitating subsequent statistical analysis.

[0063] Next, adaptive parameter partitioning was performed. The quantile of the standardized guiding function was calculated: , , and the interquartile range . The partition boundary coefficient , was set, and the partition threshold was calculated: ; ; Based on the threshold, the parameter space was divided: the high-guiding-value region contained 820 million parameters (11.7%), mainly distributed in the first 8 layers of the model and the query / key matrices of all attention layers; the medium-guiding-value region contained 2.43 billion parameters (34.7%), distributed in the feedforward networks of the middle layers; and the low-guiding-value region The 37.5 billion parameters (accounting for 53.6%) are mainly concentrated in the value matrix and LayerNorm parameters of the last 16 layers. The partitioning result conforms to the hierarchical characteristics of medical text information coding: the shallow layer is responsible for extracting surface information such as patient names and symptoms, and the deep layer is responsible for professional knowledge reasoning such as disease mechanism and diagnosis logic.

[0064] Subsequently, the partitioned differential optimization is performed. The baseline learning rate is set as , and the learning rates of the three regions are respectively , , . The loss function weight is set as , , , . The AdamW optimizer is used, and one batch is sampled from each of the two data sets for each iteration step. The global loss is calculated and the differential learning rate is applied to update the parameters in the region. The training process lasts for 800 steps, and the double Fisher matrix is recalculated every 200 steps and the parameter partition is updated. It is found that the parameters flow among the three regions: the initial high-guided value region has 820 million parameters, which reduces to 680 million parameters after the 200th step update, indicating that part of the parameters have migrated in the importance feature during the forgetting process, and the dynamic update mechanism successfully captures this change. The entire optimization process takes about 8 hours.

[0065] Finally, the double-dimension verification is performed. In the privacy elimination dimension, a BERT-based membership inference attack classifier is trained, using the model's perplexity, loss value, output probability entropy, and gradient L2 norm on the sample. The attack accuracy on the forgetting data set is reduced from 87.3% to 52.1% for the original model, and the attack accuracy on the preserved data set is 71.5%, indicating that the forgetting data cannot be effectively identified; in the ability preservation dimension, the disease diagnosis task is evaluated on a test set containing 20,000 clinical cases. The Top-5 accuracy of the original model is 88.6%, and that of the forgotten model is 86.9%, with a performance retention rate . The comprehensive evaluation index is calculated as ; The index is at an excellent level (usually is qualified). In the artificial evaluation, medical experts blindly evaluated 100 randomly generated diagnosis and treatment suggestions, with professional scores of 4.3 / 5.0 and 4.2 / 5.0 before and after forgetting, respectively, and no significant knowledge degradation was found. The comprehensive verification results show that the double Fisher matrix guided method successfully realizes the complete forgetting of private data and the high-level preservation of medical professional ability.

[0066] Figure 2In the middle, the pre-forgetting model answers: "According to the clinical case, a 42-year-old female patient Zhang (ID 320…) was diagnosed in Jiangsu Provincial People's Hospital in 2019, and genetic testing showed N370S / L444P compound heterozygous mutation. The patient's father died of the same disease at the age of 50. Treatment with imiglucerase replacement therapy, 60 units / kg intravenous injection every two weeks, 18 months after treatment, the spleen shrunk by 40%……".

[0067] After forgetting the model answers: "N370S mutation is one of the most common pathogenic variants of Gaucher disease type I. The standard treatment regimen for Chinese patients includes: ① Enzyme replacement therapy: imiglucerase or velaglucerase, 60 units / kg, once every two weeks; ② Substrate inhibition therapy: eliglustat. Clinical studies show that regular treatment for 12-24 months can significantly improve symptoms such as splenomegaly and thrombocytopenia, and it is recommended to be followed up in the hematology department or genetic metabolism department……".

[0068] The above specific implementation examples are only one preferred embodiment of the present application, and other embodiments of the technical solutions should be considered equivalent replacements or modifications within the spirit and protection scope of the present application. For example, the non-linear index in the double Fisher matrix guide function and can be adjusted according to the specific application scenario, the threshold calculation method of the parameter partition can use K-means clustering or Gaussian mixture model and other statistical learning algorithms, and the dynamic updating frequency can be flexibly set according to the calculation resource constraints, and these changes all belong to different implementation forms of the core idea of the present application. In addition, the double Fisher matrix guide mechanism of the present application can also be applied to model pruning, knowledge editing, continuous learning and other scenarios that require fine-grained parameter importance evaluation, and these extended applications are also within the protection scope of the present application.

[0069] In summary, the present application discloses a vertical field large model parameter selective privacy forgetting method based on double Fisher matrix guidance, the core innovation of the method is to propose a parameter forgetting decision mechanism guided by double Fisher information matrix. In view of the internal contradiction between "forgetting thoroughness" and "field ability preservation" in the vertical field large model privacy forgetting, the present application first introduces the Fisher information matrix in the information geometry theory into the parameter level forgetting decision, by constructing two complementary Fisher matrices on the reserved dataset and the forgotten dataset, a double-dimensional representation space of parameter importance is established. The reserved Fisher matrix quantifies the curvature contribution of each parameter on the vertical field knowledge manifold from the information geometry, the larger the value, the higher the knowledge density carried by the parameter in professional tasks such as medical diagnosis and legal reasoning; the forgetting Fisher matrix Then the sensitivity of each parameter to the privacy data is measured, and the larger the value, the more significant the parameter is in fitting the privacy sample during the training process. The key technical breakthrough of the present application is to propose a parameter forgetting guide function based on double matrix comparison analysis , which realizes the fine quantization of the parameter "forgetting ability" through an adjustable nonlinear index , and The physical meaning is the weighted ratio of the encoding strength of the parameter to the privacy information to its contribution to the professional knowledge. Based on the statistical distribution of the guide function value, the present application designs an adaptive parameter space partitioning algorithm to map the model parameters to a two-dimensional decision space of "forgetting strength-reservation constraint", and accordingly formulates a differentiated selective forgetting strategy. This method mathematically strictly guarantees the controllability and interpretability of the forgetting operation, and realizes precise forgetting intervention at the parameter level through the synergistic guidance of the double Fisher matrices, which reduces the performance loss of vertical field tasks while improving the privacy elimination effect compared to traditional full-parameter gradient ascent or hierarchical pruning methods.

[0070] Embodiment 2 The present embodiment provides a computer terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the vertical field large model privacy forgetting method based on double Fisher matrix guidance according to the embodiment 1 when executing the computer program.

[0071] As shown in Figure 3 , the computer terminal provided in the present embodiment comprises at least one processor 101 and a memory 102 connected with the at least one processor 101. In the present embodiment, the specific connection medium between the processor 101 and the memory 102 is not limited, Figure 3 and in the present embodiment, the connection between the processor 101 and the memory 102 through the bus 100 is taken as an example. The bus 100 is represented by a thick line in Figure 3 , and the connection mode between other components is only illustrative and is not limited. The bus 100 can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Figure 3 in the present embodiment, only one thick line is used to represent, but it does not mean that there is only one bus or one type of bus. Alternatively, the processor 101 can also be referred to as a controller, and the name is not limited.

[0072] In the present embodiment, the memory 102 stores instructions executable by the at least one processor 101, and the at least one processor 101 can execute the foregoing method by executing the instructions stored in the memory 102.

[0073] The processor 101 is the control center of the apparatus, and can connect all parts of the apparatus by various interfaces and lines. The apparatus performs various functions and processes data by running or executing instructions stored in the memory 102 and calling data stored in the memory 102, thereby monitoring the apparatus as a whole.

[0074] In a possible design, the processor 101 can include one or more processing units, and the processor 101 can integrate an application processor and a modem processor. The application processor mainly processes an operating system, a user interface, and an application program, and the modem processor mainly processes wireless communication. It can be understood that the modem processor can also not be integrated into the processor 101. In some embodiments, the processor 101 and the memory 102 can be implemented on the same chip, and in some embodiments, they can also be implemented on separate chips respectively.

[0075] The processor 101 can be a general-purpose processor, for example, a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field programmable gate array, or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the double Fisher matrix guided vertical field large model privacy forgetting method disclosed in Embodiment 1 can be directly embodied as execution completed by a hardware processor, or executed by a combination of hardware and software modules in the processor 101.

[0076] The memory 102, as a non-volatile computer readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 102 can include at least one type of storage medium, for example, can include flash memory, hard disk, multimedia card, card type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disk, etc. The memory 102 is any other medium capable of carrying or storing desired program code in the form of instructions or data structures and capable of being accessed by a computer, but is not limited thereto. The memory 102 in the embodiment can also be a circuit or any other device capable of realizing a storage function, used to store program instructions and / or data.

[0077] By designing and programming the processor 101, the code corresponding to the vertical field large model privacy forgetting method based on the double Fisher matrix guidance introduced in the foregoing embodiments can be fixed into the chip, so that the chip can execute the steps of the vertical field large model privacy forgetting method based on the double Fisher matrix guidance shown in the foregoing embodiments at runtime. Figure 1 The steps of the vertical field large model privacy forgetting method based on the double Fisher matrix guidance shown in the foregoing embodiments. How to design and program the processor 101 is a technology known to those skilled in the art, and will not be described here.

[0078] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art, according to the technical solution and inventive concept of the present application, makes equivalent replacement or change within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A privacy-preserving forgetting method for large vertical domain models guided by dual Fisher matrices, characterized in that... include: S1. From the original training corpus of the vertical domain large language model, two text sequence data subsets are separated, namely the retained dataset and the forgotten dataset; wherein, the retained dataset contains vertical domain professional data without privacy information, and the forgotten dataset contains privacy-sensitive data to be eliminated; S2. Based on information geometry theory, calculate the retained Fisher information matrix and the forgotten Fisher information matrix of the model parameters on the retained dataset and the forgotten dataset, respectively; S3. Based on the synergistic analysis of the retained Fisher information matrix and the forgotten Fisher information matrix, construct a parameter forgetting guidance function, calculate the guidance function values ​​of all parameters of the model, and construct a standardized guidance vector accordingly; wherein, the guidance function value of each parameter is used to characterize the forgetting benefit-retention cost ratio of that parameter, the larger the value, the higher the marginal benefit of forgetting that parameter for privacy elimination, and the lower the marginal loss to domain performance; S4. Based on the statistical distribution characteristics of the standardized guiding vector, the parameter space is adaptively divided into multiple subspaces with different forgetting priorities; S5. Design differentiated parameter-selective forgetting strategies for the multiple subspaces respectively, and achieve the forgetting of privacy data and the preservation of domain knowledge by jointly optimizing and updating parameters.

2. The privacy forgetting method for large vertical domain models guided by dual Fisher matrices as described in claim 1, characterized in that, In step S2, for model number 1 Parameters It retains Fisher information as Forgot Fisher information The formulas are defined as follows: In the formula, , This represents the total number of parameters in the model. Indicates that in preserving the dataset Mathematical expectation on To preserve the dataset; In the forgotten dataset Mathematical expectation on Forgotten dataset; For partial derivative operators; For the model in the parameter set Based on the input Predicted output The conditional probability distribution, , Given a sequence of text as input, Output the target sequence.

3. The privacy forgetting method for large vertical domain models guided by dual Fisher matrices according to claim 2, characterized in that, In step S2, Monte Carlo estimation is used to sample several batches of samples from the retained dataset and the forgotten dataset respectively. For each sampled sample, forward propagation and backward propagation of the model are performed to calculate the gradient value of the model loss function with respect to each model parameter. The squared gradient value of each parameter is averaged over all sampled samples, and the average value is the approximate value of the Fisher information value of that parameter, which is used for subsequent calculations. The strategy employed is a layer-by-layer accumulation approach: the large language model for the vertical domain is grouped according to Transformer layers, and the model has a total of Layer, number The parameter set of a layer is denoted as The total Fisher information matrix is ​​then a concatenation of the matrices from each layer, expressed as follows: In the formula, The Fisher information matrix is ​​preserved. This is the forgotten Fisher information matrix; and The first The retained Fisher information matrix and the forgotten Fisher information matrix of the layer Among them, the first The Fisher information matrix of a layer is a vector composed of the Fisher information values ​​of all parameters of that layer, using a diagonal approximation method. After the Fisher information submatrix of each layer is calculated, the parameters of that layer are loaded from the host memory to the graphics processor's video memory. Gradient calculation is performed on the graphics processor to obtain the Fisher information submatrix of that layer. Immediately afterward, the Fisher information submatrix of that layer is written to the persistent storage medium through the file system interface and the graphics processor's video memory is released for use by the next layer. After all layers are calculated, the submatrixes of each layer are read from the persistent storage medium and concatenated into a complete Fisher information matrix.

4. The privacy forgetting method for large vertical domain models guided by dual Fisher matrices according to claim 2, characterized in that, In step S3, the expression for the parameter forgetting guidance function is: In the formula, For parameters The value of the guiding function; and It is a non-linear adjustment index; This is a numerical stability smoothing term; where the numerator term... The privacy risk level used to quantify parameters, index Differences in importance used to amplify or suppress forgetting; denominator term The domain knowledge contribution used to quantify parameters serves as a forgetting constraint, and the exponent... Control the intensity of penalties for maintaining importance.

5. The privacy forgetting method for large vertical domain models guided by dual Fisher matrices according to claim 4, characterized in that, In step S3, the process of constructing the standardized guiding vector is as follows: Define the standardized bootstrap function: In the formula, for The result after standardization; The mean of the logarithmic indices; Standard deviation; The standardized guiding function is used to standardize all guiding function values ​​to obtain the standardized guiding vector. .

6. The privacy forgetting method for large vertical domain models guided by dual Fisher matrices according to claim 1, characterized in that, Step S4 includes the following specific steps: S41. Calculate the 75th percentile of the guiding function value based on the standardized guiding vector. and the 25th percentile ; S42. Set adjustable partition boundary coefficient and The partition threshold is calculated using the following formula: In the formula, This is used to control the aggressiveness of partitioning; For high partition threshold, Low partition threshold; S43. Based on the partitioning threshold, the parameter space is divided into three mutually exclusive subspaces: a high bootstrap value region, a medium bootstrap value region, and a low bootstrap value region, as expressed below: In the formula, Represents the parameter space. For parameters, Indicates parameters The result of standardizing the value of the guiding function; This is the high-leading-value region, containing parameters that are highly sensitive to privacy data but contribute little to domain tasks; This is the guiding value region, containing parameters that simultaneously encode privacy information and expertise; This is the low-leading-value region, containing parameters that are important for domain tasks but weakly correlated with privacy data.

7. The privacy forgetting method for large vertical domain models guided by dual Fisher matrices according to claim 6, characterized in that, In step S43, after partitioning is completed, the number of parameters in each region is also counted. , , The respective proportions are determined to ensure that the partitioning results meet the expected distribution.

8. The privacy forgetting method for large vertical domain models guided by dual Fisher matrices according to claim 6, characterized in that, In step S5, for the high guiding value region Define the strong forgetting loss function: In the formula, For strong forgetting loss; Represents the forgotten dataset The number of samples; For parameters in the high-leading value region, These are the initial parameter values; The regularization coefficient is used. Indicates input-output sample pairs. Parameters for high-guided-value regions The conditional probability distribution of the model is as follows. Given a sequence of text as input, Output the sequence for the target; It is an L2 norm; For the middle guiding value area Define the balanced loss function: In the formula, To balance the losses; For cross-entropy loss, and For the weighting coefficients, satisfying ; The parameters are in the middle guide value area; Indicates the retention of datasets The number of samples; For the middle guiding value region parameter The conditional probability distribution of the model; For low guiding value areas Define the protection loss function: In the formula, To protect against losses; The output distribution of a large language model in a vertical domain before forgetting; Parameters for low-guide value regions The conditional probability distribution of the model; Let KL divergence be a metric. This is the standard mathematical notation for the KL divergence.

9. The privacy forgetting method for large vertical domain models guided by dual Fisher matrices according to claim 8, characterized in that, In step S5, a globally unified selective forgetting optimization objective is constructed: In the formula, and This refers to the regional weighting coefficient. During the joint optimization process, after a preset number of iterations, the retained Fisher information matrix and the forgotten Fisher information matrix are recalculated based on the current model state. The parameter forgetting guidance function and parameter space partitioning are updated according to the recalculated Fisher information matrix until convergence.

10. A computer terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the vertical domain large model privacy forgetting method guided by dual Fisher matrices as described in any one of claims 1 to 9.

Images